fs4learning » Using neural networks for solving partial differential equations http://robotics.icstweb.org/fs4learning Just another Robotics.icstweb.org weblog Thu, 16 Apr 2009 19:51:31 +0000 en hourly 1 http://wordpress.org/?v=3.0 Network improvement. Nelder-Mead http://robotics.icstweb.org/fs4learning/2008/12/27/network-improvement-nelder-mead/ http://robotics.icstweb.org/fs4learning/2008/12/27/network-improvement-nelder-mead/#comments Sat, 27 Dec 2008 15:49:09 +0000 xni http://robotics.icstweb.org/fs4learning/?p=42 As you could see in the previous post, neural solution is quite accurately approximates analytical. But there is a problem: it’s too slow. It takes about 5 hours to have all the calculations done.

My mentor gave me an advise to use Nelder – Mead method to calculate weights. In this post I will talk about implementation of this method using F#.

My main decision was using special class to implement this method. All of the operations are done by methods inside class and the end-user only passes a functional and gets the solution that minimizes it.

My class is not a classical implementation of Nelder-Mead method, it have some improvements, which guarantee that solution at the next iteration will be better than current.

You can download source code of this class here.

With this method I got a solution in a few seconds, but it is not so good as previous.  I haven’t made any further researches for Nelder-Mead and gradient descent methods and their combinations, but I am going to do that and share my results.

Also I have an ideas:

  1. to use classification (by Kohonen maps), to sort out the worst-studied neurons
  2. to research combination of methods to adjust widths and centers
  3. to use asynchronism and parallel operations with lists to improve performance

Complete source code of my project you can find here.

]]> http://robotics.icstweb.org/fs4learning/2008/12/27/network-improvement-nelder-mead/feed/ 0 WPF & F# http://robotics.icstweb.org/fs4learning/2008/12/27/wpf-f/ http://robotics.icstweb.org/fs4learning/2008/12/27/wpf-f/#comments Fri, 26 Dec 2008 22:41:53 +0000 xni http://robotics.icstweb.org/fs4learning/?p=36 In this post I would like to write a few words about creating user interfaces to F# programs.

I don’t know about constructors for GUI, and I’m not sure about their existance at all, what’s why in this article we will create UI using objects of .NET Framework 3.0;

At first, we need to add some references to our project to make WPF available. This references are: PresentationCore, PresentationFramework and WindowsBase (all of dll’s are located at C:\Program Files\Reference Assemblies\Microsoft\Framework\v3.0\)

type ResultWindow = class
    inherit Window

Here we are creating a class called ResultWindow, that inherites defined in the framework class Window.

val mutable private minX : float

I use val keyword for declaration of the field. It’s much more simple to use mutable structures instead of immutable, but I need to notice that you shouldn’t use mutable stuctures in pure functional code. All of class members declared with val keyword must be defined in constructor.

new ( (* parameters here *) ) as this = {
        (* definition of fields *)
        minX = 0.0
                                     } then
        (* some other actions *)

Now it’s time to create a method (static methods are created without this. prefix).

member this.DrawGrid =
        let gridLine = new Line()
        (* and many-many lines of code *)

If you are not familiar to WPF my advise is to read some books about it. For example:

  • Don Syme’s WebLog
  • Practical WPF Graphics Programming — Jack Xu, Ph.D
  • WPF Recipes in C# 2008 — Sam Noble, Sam Bourton, Allen Jones

    • At the pictures presented below you can see analytic and neural solution graphics:


    I don’t publish my code in this post, because it is quite large. You can download it.

    ]]>     http://robotics.icstweb.org/fs4learning/2008/12/27/wpf-f/feed/ 0     F# and WPF http://robotics.icstweb.org/fs4learning/2008/12/01/f-and-wpf/ http://robotics.icstweb.org/fs4learning/2008/12/01/f-and-wpf/#comments Mon, 01 Dec 2008 12:40:28 +0000 xni http://robotics.icstweb.org/fs4learning/?p=33 By the moment I have a solution of the problem: I have the simplest artificial neural network that solves elliptic equation. For further research process I need a visualization of my solution.

    There are three way to have this done:

    • Text output into file and writing application on other language to show the results. This is a fast and simple solution. But my F# program returns list of weights, widths and centers. So I need to write the neural network modeler again! Oh, no… I don’t like this way.
    • Use the ability of Microsoft Visual Studio to call procedures, written on other languages inside one project. At first I wanted to use this ability, but yesterday I find the best way to do the modeler.
    • Windows Presentation Foundation (WPF). It’s a part of .NET framework 3 and gives us an opportunity to have separate code and markup written.

    I need to have my visualizer finished in 3 days, so results are coming soon!

    There are a lot of materials about F# and WPF, but I liked this one.

    ]]>     http://robotics.icstweb.org/fs4learning/2008/12/01/f-and-wpf/feed/ 0     Neural network overview http://robotics.icstweb.org/fs4learning/2008/11/10/neural-network-overview/ http://robotics.icstweb.org/fs4learning/2008/11/10/neural-network-overview/#comments Sun, 09 Nov 2008 23:00:25 +0000 xni http://robotics.icstweb.org/fs4learning/?p=18 If you still don’t know what the artificial neural network is -  you should read some information about it (e.g. http://en.wikipedia.org/wiki/Artificial_neuron).

    In this article I will use artificial neurons based on radial basis functions (RBF). The influence of a neuron located in point c upon x is calculated as: \varphi(x) = e^{\frac{||x-c||^2}{a^2}} where a is also a parameter of current neuron. Let us consider that the network consists on N neurons. Then we are able to calculate total influence: u(x) = \sum\limits_{i=1}^N w_i \varphi_i(x)=\sum\limits_{i=1}^N w_i e^{\frac{||x-c_i||^2}{a_i^2}}

    The first partial differential equation that I’ll solve is: \frac{\partial^2 U}{\partial x^2} + \frac{\partial^2 U}{\partial y^2} = 1 U(0) = 0 U(1) = 0


    At first iteration I define sets of neuron centers (funCenters), widths (funWidths), and collocation points. A collocation point is a point, where total influence of all neurons will be calculated, compared to a given value, and after that’s done, parameter corrections will be made. There are two types of collocation points: points located inside the area and points lying on it’s border.

    (* Randomly selected values *)
let funCenters = [-0.3; 0.1; 0.15; 0.2; 0.3; 0.45; 0.5; 0.52; 0.6; 0.69; 0.8; 0.91; 1.03]
(* At first I suggest what width is 0.3 *)
let funWidths = List.map (fun x -> 0.3) [1..13]
(* Inner collocation points *)
let collocationInside = [0.01; 0.2; 0.25; 0.41; 0.56; 0.61; 0.78; 0.8; 0.95; 0.99]
(* Border collocation points *)
let collocationOutside = [0.0; 1.0]

    I declare some helpful functions:

    (* calculate a^b *)
let rec pow (a:float) (b:int) =
    match b with
    | 0 -> 1.0
    | _ -> a*( pow a (b-1) )
 
(* generate a RBF with specified center and width *)
let funcGen (center:float) (width:float) =
    fun pos -> exp (-(pow (pos-center) 2)/(pow width 2))
 
(* generate a second derivate (d^2 U / d x^2) for RBF with specified center and width *)
let ddfuncGen (center:float) (width:float)=
    fun pos -> ( 4.0 * (pow (pos - center) 2) / (pow width 4) - 2.0/(pow width 2)) * exp (-(pow (pos-center) 2) / (pow width 2))
 
(* right part of PDE *)
let neodn (x:float) =
    1.0
 
(* border values of PDE *)
let funcBorder (x:float) =
    match x with
    | 0.0 -> 0.0
    | 1.0 -> 0.0
    | _ -> 0.0

    Now I need to fill a list of weights. To do that I use LSM and the dnAnalytics library.

    (* first approximation of weights using LSM *)
let getFirstWeights =
    (* list of RBF functions, generated for each center-width pair by funcGen function *)
    let phiList = List.map2 funcGen funCenters funWidths
    (* list of second derivatives for RBF for each center-width pair *)
    let ddphiList = List.map2 ddfuncGen funCenters funWidths
    (* LSM *)
    (* every row i of H represents i-th collocation point, and every value j in i-th row is a contribution of j-th neuron. Z[i] = right part of PDE for inner collocation points i or border value for border collocation point *)
    let mutable H = dnAnalytics.LinearAlgebra.DenseMatrix((collocationInside |> List.length) + (collocationOutside |> List.length), funCenters |>List.length)
    let mutable Z = DenseVector((collocationInside |> List.length) + (collocationOutside |> List.length))
    for i=0 to (collocationInside |> List.length)-1 do
        for j=0 to (funCenters |>List.length)-1 do
            H.Item(i,j) <- ddphiList.[j] collocationInside.[i]
        done
        Z.Item(i)  List.length)-1 do
        for j=0 to (funCenters |>List.length)-1 do
            H.Item(i+(collocationInside |> List.length),j) <- phiList.[j] collocationOutside.[i]
        done
        Z.Item(i) <- funcBorder collocationOutside.[i]
    done
    let W = ((H.Transpose())*H).Inverse()
    let w = W*(H.Transpose())*Z
    (* you can find vectorToList function into source file *)
    vectorToList w 0

    After a priori definition of weights I begin network training. I do this by fixing all parameters except one and minimizing residual value I (N — number of inner collocation points, M — number of border collocation points, K — number of neurons): I = \sum\limits_{i=1}^{N} \left(\left(\sum\limits_{k=1}^{K} w_k\frac{\partial^2 \varphi_k(x_i)}{\partial x^2}\right) - f(x_i)\right)^2 + \lambda\sum\limits_{j=1}^{M}\left(\left(\sum\limits_{k=1}^{K} w_k \varphi_k(x_j)\right)-b(x_j)\right)^2

    Minimisation of I is made by gradient descent method so we need to calculate all partial derivatives \frac{\partial I}{\partial a_i}\qquad \frac{\partial I}{\partial c_i}\qquad \frac{\partial I}{\partial w_i}

    (* function sum3 gets 3 lists and performs an operation f over heads of all lists, adding result into acc *)
let rec sum3 (f : float -> float -> float -> float -> float) (a : float list) (b : float list) (c : float list) x acc =
    match a with
    | [] -> acc
    | _ -> sum3 f (List.tl a) (List.tl b) (List.tl c) x (acc+(f (List.hd a) (List.hd b) (List.hd c) x))
 
(* training network *)
let solve =
    (* get weights for first iteration by LSM *)
    let weights = getFirstWeights
    (* loop *)
    let rec fo a b (w : float list) (az : float list) (fc : float list) =
        match a with
        | _ when a< b ->
                        (* residual for some inner point x (l1 -- weights, l2 -- widths, l3 -- centers list *)
                        let residualDerivative l1 l2 l3 x =
                            let rez = (sum3 (fun w a c x -> w * (4.0*(x-c)*(x-c)-2.0*a*a)/(a*a*a*a)*(exp (-(x-c)*(x-c)/(a*a)))) l1 l2 l3 x 0.0) - (neodn x)
                            rez
                        (* residual for some border point x *)
                        let residual l1 l2 l3 x =
                            let rez = (sum3 (fun w a c x -> w * (exp (-(x-c)*(x-c)/(a*a)))) l1 l2 l3 x 0.0) - (funcBorder x)
                            rez
                        (* dI/dw for neuron with width a and center c *)
                        let dIw (a : float) (c : float) =
                            let lambda = 100.0
                            let rez = (List.sum_by (fun x -> 2.0*(residualDerivative w az fc x)*(4.0*(x-c)*(x-c)-2.0*a*a)/(a*a*a*a)*(exp (-(x-c)*(x-c)/(a*a)))) collocationInside ) + (List.sum_by (fun x -> 2.0*lambda*(residual w az fc x)*(exp (-(x-c)*(x-c)/(a*a)))) collocationBorder)
                            rez
 
                        (* generate list of derivatives for current widths az and centers fc *)
                        let dIdw = List.map2 dIw az fc
                        (* update widths by gradient descent method. Step is fixed (I am planning to add adaptive step) *)
                        let weights2 = List.map2 (fun a b -> a-(1e-12)*b) w dIdw
 
                        (* dI/da  for neuron with width a, center c and weight w *)
                        let dIa (w : float) (a : float) (c : float) =
                            let lambda = 100.0
                            let rez = (List.sum_by (fun x -> 2.0*(residualDerivative weights2 az fc x)*w*4.0*(exp (-(x-c)*(x-c)/(a*a))*(a*a*a*a-5.0*(x-c)*(x-c)*a*a+2.0*(pow (x-c) 4))/(pow a 7))) collocationInside) + (List.sum_by (fun x -> 2.0*lambda*(residual weights2 az fc x)*w*2.0*(x-c)*(x-c)/(a*a*a)*(exp (-(x-c)*(x-c)/(a*a)))) collocationBorder)
                            rez
 
                        (* generate list of derivatives for current widths2, centers fc, widths az *)
                        let dIda = List.map3 dIa weights2 az fc
 
                        (* update widths *)
                        let az2 = List.map2 (fun a b -> a-(1e-12)*b) az dIda
 
                        (* dI/dc  for neuron with width a, center c and weight w *)
                        let dIс (w : float) (a : float) (c : float) =
                            let lambda = 100.0
                            let rez = (List.sum_by (fun x -> 2.0*(residualDerivative weights2 az2 fc x)*w*(x-c)*(-12.0*a*a+8.0*(x-c)*(x-c))/(pow a 6)*(exp (-(x-c)*(x-c)/(a*a)))) collocationInside) + (List.sum_by (fun x -> 2.0*lambda*(residual weights2 az2 fc x)*w*2.0*(x-c)/(a*a)*(exp (-(x-c)*(x-c)/(a*a)))) collocationBorder)
                            rez
 
                        (* generate list of derivatives for current widths2, centers fc, widths az2 *)
                        let dIdc = List.map3 dIс weights2 az2 fc
 
                        (* update centers *)
                        let fc2 = List.map2 (fun a b -> a-(1e-12)*b) fc dIdc
 
                        fo (a+1) b weights2 az2 fc2
        | _ -> ()
    fo 0 1000 weights funWidths funCenters
    (* now I make 1000 of iterations what is enought. But in the future I will change this condition *)
    (* return true *)
    true

    This model is made for learning purposes and I think that now it’s very simple to analyze parameters and build a more complicated solution.

    As always, source code is available.

    ]]>     http://robotics.icstweb.org/fs4learning/2008/11/10/neural-network-overview/feed/ 0