Applications of Neural Networks to Music Generation - Revision history

User at 03:40, 22 October 2022

2022-10-22T03:40:57Z

User at 03:40, 22 October 2022

2022-10-22T03:40:57Z

User: /* Ongoing Projects – MuseNet */

2022-10-22T03:40:57Z

‎Ongoing Projects – MuseNet

User: /* Ongoing Projects – MuseNet */

2022-10-22T03:40:57Z

‎Ongoing Projects – MuseNet

User: /* Ongoing Projects – MuseNet */

2022-10-22T03:40:56Z

‎Ongoing Projects – MuseNet

User: /* Long Short-Term Memory Cells (LSTMs) */

2022-10-22T03:40:55Z

‎Long Short-Term Memory Cells (LSTMs)

User: /* Long Short-Term Memory Cells (LSTMs) */

2022-10-22T03:40:55Z

‎Long Short-Term Memory Cells (LSTMs)

User: /* Long Short-Term Memory Cells (LSTMs) */

2022-10-22T03:40:55Z

‎Long Short-Term Memory Cells (LSTMs)

User: /* Long Short-Term Memory Cells (LSTMs) */

2022-10-22T03:40:54Z

‎Long Short-Term Memory Cells (LSTMs)

User: /* Recurrent Neural Network Cells (RNNs) */

2022-10-22T03:40:54Z

‎Recurrent Neural Network Cells (RNNs)

← Older revision		Revision as of 03:40, 22 October 2022
Line 1:		Line 1:
−	By Anna Mikhailova	+	'''By Anna Mikhailova'''

	Music generation, alternatively termed algorithmic composition, or music synthesis, is the process of generating music via an algorithm (i.e. some formal set of rules). Algorithmic music composition has been a part of the human experience for millennia – transcending cultures and continents. It ranges from any music that is generated without human interference – e.g. wind chimes – to music generated by intelligent systems. The application of neural networks to music generation poses particular interest due to the similarity between music composition and natural language structuring. A similar challenge to that of language is that any given instance of “music” can contain more than just one music unit (i.e. a note). It can consist of several notes together (a chord) and/or other qualities such as volume, much like language. The creation of “convincing” music, or music that sounds like it could be naturally produced by a human, is riddled with challenges. Although not all algorithmic music has aimed to produce human-like creations, that goal has been at the forefront of ongoing research.		Music generation, alternatively termed algorithmic composition, or music synthesis, is the process of generating music via an algorithm (i.e. some formal set of rules). Algorithmic music composition has been a part of the human experience for millennia – transcending cultures and continents. It ranges from any music that is generated without human interference – e.g. wind chimes – to music generated by intelligent systems. The application of neural networks to music generation poses particular interest due to the similarity between music composition and natural language structuring. A similar challenge to that of language is that any given instance of “music” can contain more than just one music unit (i.e. a note). It can consist of several notes together (a chord) and/or other qualities such as volume, much like language. The creation of “convincing” music, or music that sounds like it could be naturally produced by a human, is riddled with challenges. Although not all algorithmic music has aimed to produce human-like creations, that goal has been at the forefront of ongoing research.

← Older revision		Revision as of 03:40, 22 October 2022
Line 1:		Line 1:
	By Anna Mikhailova		By Anna Mikhailova
−
−	~~'''Note: this page is incomplete.'''~~

	Music generation, alternatively termed algorithmic composition, or music synthesis, is the process of generating music via an algorithm (i.e. some formal set of rules). Algorithmic music composition has been a part of the human experience for millennia – transcending cultures and continents. It ranges from any music that is generated without human interference – e.g. wind chimes – to music generated by intelligent systems. The application of neural networks to music generation poses particular interest due to the similarity between music composition and natural language structuring. A similar challenge to that of language is that any given instance of “music” can contain more than just one music unit (i.e. a note). It can consist of several notes together (a chord) and/or other qualities such as volume, much like language. The creation of “convincing” music, or music that sounds like it could be naturally produced by a human, is riddled with challenges. Although not all algorithmic music has aimed to produce human-like creations, that goal has been at the forefront of ongoing research.		Music generation, alternatively termed algorithmic composition, or music synthesis, is the process of generating music via an algorithm (i.e. some formal set of rules). Algorithmic music composition has been a part of the human experience for millennia – transcending cultures and continents. It ranges from any music that is generated without human interference – e.g. wind chimes – to music generated by intelligent systems. The application of neural networks to music generation poses particular interest due to the similarity between music composition and natural language structuring. A similar challenge to that of language is that any given instance of “music” can contain more than just one music unit (i.e. a note). It can consist of several notes together (a chord) and/or other qualities such as volume, much like language. The creation of “convincing” music, or music that sounds like it could be naturally produced by a human, is riddled with challenges. Although not all algorithmic music has aimed to produce human-like creations, that goal has been at the forefront of ongoing research.

@@ Line 79: / Line 79: @@
 Today, there are many ongoing projects in the field of Algorithmic Music. One of particular note is MuseNet <ref name= "musenet">Payne, Christine. "MuseNet." OpenAI, 25 Apr. 2019, openai.com/blog/musenet</ref>. MuseNet is a deep neural network system that generates 4-minute-long musical compositions with up to ten musical instruments. MuseNet implements a large-scale, general-purpose, unsupervised, transformer model <ref>Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017). Attention Is All You Need. arXiv. https://doi.org/10.48550/arXiv.1706.03762</ref>, trained to predict the next token in a sequence. The transformer model is based on Recurrent Neural Networks, but takes it a step further, in that it is based solely on attention mechanisms to draw dependencies between input and output. This model allows for significantly more parallelization than previous mechanisms by using stacked self-attention and point-wise, fully connected layers for both the encoder and decoder. The MuseNet system built on the transformer generates quite convincing compositions that indeed sound like they could have been produced by a human. Below are several examples:
-<iframe width="100%" height="166" scrolling="no" frameborder="no" allow="autoplay" src="https://w.soundcloud.com/player/?url=https%3A//api.soundcloud.com/tracks/609806943&color=%23ff5500&auto_play=false&hide_related=false&show_comments=true&show_user=true&show_reposts=false&show_teaser=true"></iframe><div style="font-size: 10px; color: #cccccc;line-break: anywhere;word-break: normal;overflow: hidden;white-space: nowrap;text-overflow: ellipsis; font-family: Interstate,Lucida Grande,Lucida Sans Unicode,Lucida Sans,Garuda,Verdana,Tahoma,sans-serif;font-weight: 100;"><a href="https://soundcloud.com/openai_audio" title="OpenAI" target="_blank" style="color: #cccccc; text-decoration: none;">OpenAI</a> · <a href="https://soundcloud.com/openai_audio/chopin-f-minor-etude" title="Prompt: First 5 notes of Chopin Op. 10, No. 9" target="_blank" style="color: #cccccc; text-decoration: none;">Prompt: First 5 notes of Chopin Op. 10, No. 9</a></div>
+[https://soundcloud.com/openai_audio/chopin-f-minor-etude Prompt: First 5 notes of Chopin Op. 10, No. 9]
 == References ==
 <references />

@@ Line 77: / Line 77: @@
 == Ongoing Projects – MuseNet ==
-Today, there are many ongoing projects in the field of Algorithmic Music. One of particular note is MuseNet <ref name= "musenet">Payne, Christine. "MuseNet." OpenAI, 25 Apr. 2019, openai.com/blog/musenet</ref>. MuseNet is a deep neural network system that generates 4-minute-long musical compositions with up to ten musical instruments. MuseNet implements a large-scale, general-purpose, unsupervised, transformer model <ref>Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017). Attention Is All You Need. arXiv. https://doi.org/10.48550/arXiv.1706.03762</ref>, trained to predict the next token in a sequence. The transformer model is based on Recurrent Neural Networks, but takes it a step further, in that it is based solely on attention mechanisms to draw dependencies between input and output. This model allows for significantly more parallelization than previous mechanisms by using stacked self-attention and point-wise, fully connected layers for both the encoder and decoder. The MuseNet system built on the transformer generates quite convincing compositions that indeed sound like they could have been produced by a human.
+Today, there are many ongoing projects in the field of Algorithmic Music. One of particular note is MuseNet <ref name= "musenet">Payne, Christine. "MuseNet." OpenAI, 25 Apr. 2019, openai.com/blog/musenet</ref>. MuseNet is a deep neural network system that generates 4-minute-long musical compositions with up to ten musical instruments. MuseNet implements a large-scale, general-purpose, unsupervised, transformer model <ref>Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017). Attention Is All You Need. arXiv. https://doi.org/10.48550/arXiv.1706.03762</ref>, trained to predict the next token in a sequence. The transformer model is based on Recurrent Neural Networks, but takes it a step further, in that it is based solely on attention mechanisms to draw dependencies between input and output. This model allows for significantly more parallelization than previous mechanisms by using stacked self-attention and point-wise, fully connected layers for both the encoder and decoder. The MuseNet system built on the transformer generates quite convincing compositions that indeed sound like they could have been produced by a human. Below are several examples:
 == References ==
 <references />

@@ Line 77: / Line 77: @@
 == Ongoing Projects – MuseNet ==
 == References ==
 <references />

@@ Line 68: / Line 68: @@
 == Long Short-Term Memory Cells (LSTMs) ==
-To combat the vanishing gradient problem of RNNs, modern language learning systems and algorithmic music systems implement Long Short-Term Memory <ref>Hochreiter, S. (1997). LONG SHORT-TERM MEMORY. Neural Computation, 9(8). Retrieved November 18, 2022, from http://www.bioinf.jku.at/publications/older/2604.pdf</red> cells (LSTMs), first introduced by Sepp Hochreiter in 1997. As the error is propagated throughout the RNN, hidden layers are connected to themselves in times by means of recurrent weights (w_rec). Because the recurrent weight is compounded, we witness the vanishing gradient effect. Numerically speaking, the vanishing gradient problem is present if the recurrent weight achieves a value less than 1 (greater than 1 creates an exploding gradient problem [https://analyticsindiamag.com/can-relu-cause-exploding-gradients-if-applied-to-solve-vanishing-gradients/#:~:text=The%20exploding%20gradient%20is%20the,learning%20from%20your%20training%20data.]). In a gross oversimplification, Long Short-Term Memory cells make the recurrent weight equal to 1.
+To combat the vanishing gradient problem of RNNs, modern language learning systems and algorithmic music systems implement Long Short-Term Memory <ref>Hochreiter, S. (1997). LONG SHORT-TERM MEMORY. Neural Computation, 9(8). Retrieved November 18, 2022, from http://www.bioinf.jku.at/publications/older/2604.pdf</ref> cells (LSTMs), first introduced by Sepp Hochreiter in 1997. As the error is propagated throughout the RNN, hidden layers are connected to themselves in times by means of recurrent weights (w_rec). Because the recurrent weight is compounded, we witness the vanishing gradient effect. Numerically speaking, the vanishing gradient problem is present if the recurrent weight achieves a value less than 1 (greater than 1 creates an exploding gradient problem [https://analyticsindiamag.com/can-relu-cause-exploding-gradients-if-applied-to-solve-vanishing-gradients/#:~:text=The%20exploding%20gradient%20is%20the,learning%20from%20your%20training%20data.]). In a gross oversimplification, Long Short-Term Memory cells make the recurrent weight equal to 1.
 A brief reminder of the structure of an RNN – the hidden layer receives an input x<sub>t</sub> from the input layer at time t, and from itself at time t-1, and it generates an output h<sub>t</sub> at time t and also another input for itself at time t+1. To solve the vanishing gradient problem, the structure of an RNN is transformed into an LSTM as follows <ref name= "LSTM" />:

@@ Line 74: / Line 74: @@
 # Every block also produces two vector outputs: an output h<sub>t</sub> and c<sub>t</sub>
-In 2002, Doug Eck, currently head of the Magenta team at Google Brain, used LSTM cells to create a system that improvised blues music based on a data set of short recordings. The publication is linked here <ref>Eck, D., &amp; Schmidhuber, J. (2002). Finding Temporal Structure in Music: Blues Improvisation with LSTM Recurrent Networks. Neural Networks for Signal Processing XII, Proceedings of the 2002 IEEE Workshop, 747-756. Retrieved November 18, 2022, from http://www.iro.umontreal.ca/~eckdoug/papers/2002_ieee.pdf. </ref>.
+In 2002, Doug Eck, currently head of the Magenta team at Google Brain, used LSTM cells to create a system that improvised blues music based on a data set of short recordings. The publication is linked here <ref> Eck, D., &amp; Schmidhuber, J. (2002). Finding Temporal Structure in Music: Blues Improvisation with LSTM Recurrent Networks. Neural Networks for Signal Processing XII, Proceedings of the 2002 IEEE Workshop, 747-756. Retrieved November 18, 2022, from http://www.iro.umontreal.ca/~eckdoug/papers/2002_ieee.pdf.</ref>.
 == Ongoing Projects – MuseNet ==

@@ Line 44: / Line 44: @@
 # Take a set (the larger, generally the better) of music data, i.e. compositions, with each composition represented as a sequence of chords with properties of pitch, note, octave, frequency, and amplitude.
-# Go through the sequence steps and calculate the probability distribution of every possible chord, given the chords occurring so far (computing the output distribution ŷ(t) for every step t.
+# Go through the sequence steps and calculate the probability distribution of every possible chord, given the chords occurring so far (computing the output distribution ŷ<sup>(t)</sup> for every step t.
 # At every step t, compute the loss function – the cross-entropy between the predicted probability distribution and the true next chord.
 # Take the average of all of the loss functions to compute the overall loss for the entire training set.