Hi, thank you so much for looking into my Kickstarter!
Who Am I:
I'm a Neuroscience Graduate student at Brandeis University who has recently become extremely fascinated by the field of Machine-Learning, especially Artificial Neural Networks, the most promising form of A.I. for many tasks. I've become passionate about the idea of applying principles of real neurons (such as the ability to switch between firing modes or the ability to adjust sensitivity to input) to the nodes in Long-Short-Term-Memory cells. Long-Short-Term-Memory cells are a form of recurrent neural networks that was, itself, inspired by neuroscience (neurons in the Hippocampus to be precise), so I believe that's the perfect type of network to test my hypothesis on.
I'm hoping that by applying certain principles of real neurons, to Artificial Neural Networks, I will be able to improve these Networks' accuracy and generalization, thereby demonstrating the need for more research applying the cellular mechanisms of real neurons to Artificial Neural Networks. By the end of the upcoming year of University, with your help, I'm hoping to have completed a Master's Thesis, using this data.
What Am I Doing:
I'm going to be running a tremendous amount of experiments to ensure that any positive results I get are reliable, and have the proper controls. Preliminary data looks good, with statistical significance all around, but the sample size is small (N=40), and not every necessary control has been tested. It took approximately 2 months of continuous code-running to collect the preliminary data on my laptop, and the full dataset I need to collect is about 150 times as large (8x the sample size, 20x the number of conditions).To add some perspective, it took approximately an hour to train a network once/sample size of 1, and I am aiming now for a sample size of 200, per condition.)
What Is The Exact Hypothesis:
Akin to how real neurons have the ability to switch between firing mode, and neurotransmitter release, I have coded in a layer of nodes (which could be seen as analogous to epigenetic expression) which choose between (the two most commonly used activation functions), Tanh and ReLU, at each time-step, for each node of the core layer of the LSTM cell. (An activation function tells the neuron the relationship between its input and output, Tanh is a sigmoid activation which translates any input into a value between -1 and 1, ReLU is a linear activation function which translates input into an output between 0 and infinity). Thereby my modifications provide these nodes the capability to switch between two modes of activation. Similar to a neuron switching between a tonic and bursting firing mode.
Additionally, akin to how real neurons can release more or less transmitter in response to the same stimuli, due to factors such as facilitation and depression, I have also added a layer that scales the slope of the activation function of the core layer nodes, at each time-step, for each node. This enables a dynamic sensitivity to input that can be envisioned as the intrinsic excitability of the neuron or axon terminal modulation (facilitation & depression). This alone, and this in combination with the previous condition, both yielded positive preliminary results, and have the potential to become standard practice if further data-collection demonstrates that the preliminary results are reliable.
Why Do I Need The Money:
Because there is so much data collect, the only feasible option appears to be renting a bunch of GPUs on the Amazon or Google Cloud. This happens to be extremely expensive, and since I am not a PhD student, my research is not funded by my University. Therefore, in order to continue my research, my options are to either pay out of pocket, get a grant from Amazon (fingers crossed), or hope that the internet gods smile upon this Kickstarter.
When running the preliminary data on my laptop with 1 GPU and 4 CPU's, training the network once took about an hour, therefore, a sample size of 1 requires an hour, given 1 GPU. For each experimental condition (of which there are 2), there are about 16 controls, all needing a sample size of 200. I estimated, that in order to shrink the estimated run-time for code down from 3,200 hours (132 days), to 200 hours (9 days), it would require 16 GPU's, which is available for about 500$ a day (4,166$ total). Since I have 2 main conditions I would like to test, this will likely require 8,333$, and then if either condition gets glowing results, I must repeat the process with 2 more sets of training data, in order to ensure reliability across data-sets. This lands me at about 25,000$ for every experimental condition to be fully tested.
But the amount I set for this Kickstarter should be enough to test the first experimental condition. However if, by way of magic, 25,000$ is actually raised, it will go towards running the full set of experimental conditions on the Cloud. If Elon Musk stumbles upon this Kickstarter and decides to donate more than 25,000$, every extra thousand will go towards reserving more GPU's, such that I can increase the sample size even further, with the ultimate goal of reaching a sample size of 1,000 per condition. Currently, my preliminary sample size of 40 is laughable, with a sample size of 200, my results will be noteworthy, but if 100,000$ is raised, the resulting sample size of 1,000 could make my research actually publishable.
After this data is collected, I plan to write a master's thesis on it, or ideally, publish this data in a reputable Journal. Additionally, I plan to continue on with another hypothesis, regarding the connection between Artificial Neural Networks' Learning Rate, and the Plasticity of synapses. I believe that the current standard practice of having a set Learning Rate for every node at each Epoch may be analogous to Critical Periods of high plasticity. However, after the initial period of development, the accuracy of Artificial Neural Networks tapers off sharply and requires dramatic decreases to the Learning Rate of the nodes, which too, mimics neural development. What seems, to me, to be missing, is the fact that after early human development, while the human brain is not nearly as plastic as it was, there is still an enormous amount of plasticity which occurs conditionally on a synapse-by-synapse basis due to dynamic factors such as Activity-Dependent Plasticity. If my current project is completed within the first Semester, I would like to expand my model of Neuro-realistic nodes, by enabling node-specific learning rates which follow a set of rules based off certain mechanisms by which synaptic plasticity works in the human brain.
Risks and challenges
The first risk regards the Cloud computing speed. If the Cloud computing speed is slowly or faster than estimated, than this could cause drastic changes to the what can be achieved with the raised funds. On the optimistic side, I may have underestimated their power, and will therefore be able to increase my sample size. On the pessimistic side, the raised funds may only be able to pay for a smaller sample size than I hoped for.
Additionally, there is always the risk in research, that the results simply come out non-significant. I want this money to be able to test my hypothesis by collecting a tremendous amount of data using many GPU's. But even after collecting the data, there is a significant possibility that my hypothesis will not be supported.
Preliminary data looks very good, but it's only preliminary, and I don't want anyone funding me with the idea that this is a guaranteed way to revolutionize Machine Learning.
If my research comes back positive, than this will be a small step forward in demonstrating the potential of this path in Machine-Learning research. But if the research comes back with poor results, I will have to move on to subsequent hypotheses. While I already have a hypothesis to test next (applying the principle of synpatic plasticity to the learning rate of nodes), it has not yet been coded and it too could fail.
I heard an amazing quote about research once. Research is about searching, and you're going to fail, alot, that is why it's called re-search.
Learn about accountability on Kickstarter