Comparison and Analysis of Neural Solver Methods for Differential Equations in Physical Systems
DOI:
https://doi.org/10.26418/elkha.v13i2.49097Keywords:
Differential Equations, Deep Learning, Neural Networks, Numerical Methods.Abstract
Differential equations are ubiquitous in many fields of study, yet not all equations, whether ordinary or partial, can be solved analytically. Traditional numerical methods such as time-stepping schemes have been devised to approximate these solutions. With the advent of modern deep learning, neural networks have become a viable alternative to traditional numerical methods. By reformulating the problem as an optimisation task, neural networks can be trained in a semi-supervised learning fashion to approximate nonlinear solutions. In this paper, neural solvers are implemented in TensorFlow for a variety of differential equations, namely: linear and nonlinear ordinary differential equations of the first and second order; Poisson"™s equation, the heat equation, and the inviscid Burgers"™ equation. Different methods, such as the naive and ansatz formulations, are contrasted, and their overall performance is analysed. Experimental data is also used to validate the neural solutions on test cases, specifically: the spring-mass system and Gauss"™s law for electric fields. The errors of the neural solvers against exact solutions are investigated and found to surpass traditional schemes in certain cases. Although neural solvers will not replace the computational speed offered by traditional schemes in the near future, they remain a feasible, easy-to-implement substitute when all else fails.
References
F. Rosenblatt, “The perceptron: A probabilistic model for information storage and organization in the brain,†Psychol. Rev., vol. 65, no. 6, pp. 386–408, Nov. 1958, doi: 10.1037/h0042519.
D. E. Rumelhart, G. E. Hinton, and R. J. Williams, “Learning representations by back-propagating errors,†Nature, vol. 323, no. 6088, pp. 533–536, 1986, doi: 10.1038/323533a0.
Y. Lecun, Y. Bengio, and G. Hinton, “Deep learning,†Nature, vol. 521, no. 7553, pp. 436–444, May 2015, doi: 10.1038/nature14539.
K. He, X. Zhang, S. Ren, and J. Sun, “Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification,†in 2015 IEEE International Conference on Computer Vision (ICCV), 2015, pp. 1026–1034, doi: 10.1109/ICCV.2015.123.
A. Vaswani et al., “Attention is all you need,†in Advances in Neural Information Processing Systems, Jun. 2017, vol. 2017-Decem, pp. 5999–6009, Accessed: Jan. 28, 2021. [Online]. Available: https://arxiv.org/abs/1706.03762v5.
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,†NAACL HLT 2019 - 2019 Conf. North Am. Chapter Assoc. Comput. Linguist. Hum. Lang. Technol. - Proc. Conf., vol. 1, pp. 4171–4186, Oct. 2018, Accessed: Jan. 28, 2021. [Online]. Available: http://arxiv.org/abs/1810.04805.
M. W. M. G. Dissanayake and N. Phan-Thien, “Neural-network-based approximations for solving partial differential equations,†Commun. Numer. Methods Eng., vol. 10, no. 3, pp. 195–201, Mar. 1994, doi: 10.1002/cnm.1640100303.
I. E. Lagaris, A. Likas, and D. I. Fotiadis, “Artificial neural networks for solving ordinary and partial differential equations,†IEEE Trans. Neural Networks, vol. 9, no. 5, pp. 987–1000, 1998, doi: 10.1109/72.712178.
J. Berg and K. Nyström, “A unified deep artificial neural network approach to partial differential equations in complex geometries,†Neurocomputing, vol. 317, pp. 28–41, Nov. 2018, doi: 10.1016/j.neucom.2018.06.056.
J. Sirignano and K. Spiliopoulos, “DGM: A deep learning algorithm for solving partial differential equations,†J. Comput. Phys., vol. 375, pp. 1339–1364, Dec. 2018, doi: 10.1016/j.jcp.2018.08.029.
G. Cybenko, “Approximation by superpositions of a sigmoidal function,†Math. Control. Signals, Syst., vol. 2, no. 4, pp. 303–314, Dec. 1989, doi: 10.1007/BF02551274.
S. Ruder, “An overview of gradient descent optimization algorithms,†arXiv, Sep. 2016, Accessed: Jan. 28, 2021. [Online]. Available: http://arxiv.org/abs/1609.04747.
M. L. Piscopo, M. Spannowsky, and P. Waite, “Solving differential equations with neural networks: Applications to the calculation of cosmological phase transitions,†Phys. Rev. D, vol. 100, no. 1, Jul. 2019, doi: 10.1103/PhysRevD.100.016002.
M. Baymani, S. Effati, H. Niazmand, and A. Kerayechian, “Artificial neural network method for solving the Navier–Stokes equations,†Neural Comput. Appl., vol. 26, no. 4, pp. 765–773, May 2015, doi: 10.1007/s00521-014-1762-2.
W. Tang et al., “Study on a Poisson’s equation solver based on deep learning technique,†in 2017 IEEE Electrical Design of Advanced Packaging and Systems Symposium, EDAPS 2017, Jan. 2018, vol. 2018-Janua, pp. 1–3, doi: 10.1109/EDAPS.2017.8277017.
M. Abadi et al., “TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems,†CoRR, vol. abs/1603.0, Mar. 2016, Accessed: Jan. 28, 2021. [Online]. Available: http://arxiv.org/abs/1603.04467.
M. Abadi et al., “TensorFlow: A system for large-scale machine learning,†in Proceedings of the 12th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2016, May 2016, pp. 265–283, Accessed: Jan. 28, 2021. [Online]. Available: http://arxiv.org/abs/1605.08695.
D. P. Kingma and J. L. Ba, “Adam: A method for stochastic optimization,†Dec. 2015, Accessed: Jan. 28, 2021. [Online]. Available: https://arxiv.org/abs/1412.6980v9.
G. Moradi and M. Marvasti, “Experimental Solution to the Laplace Equation, a Tutorial Approach,†IJARCCE, vol. 5, no. 9, pp. 278–284, Sep. 2016, doi: 10.17148/ijarcce.2016.5960.
R. Courant, K. Friedrichs, and H. Lewy, “Über die partiellen Differenzengleichungen der mathematischen Physik,†Math. Ann., vol. 100, no. 1, pp. 32–74, Dec. 1928, doi: 10.1007/BF01448839.
Downloads
Additional Files
Published
Issue
Section
License
Copyright (c) 2021 ELKHA : Jurnal Teknik Elektro
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
1. Proposed Policy for Journals That Offer Open Access
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).
2. Proposed Policy for Journals That Offer Delayed Open Access
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication, with the work [SPECIFY PERIOD OF TIME] after publication simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).
