At mDecks Music we develop apps to study and practice music and music theory, mostly for intermediate to advanced musicians.

Our apps need to interact with the user in many ways: from playing simple chords using sampled sounds to complex accompaniments, received and send MIDI data and also listen to audio input from the user and convert into useful musical information.

Without a doubt, using AudioKit allowed us to achieve all these tasks easily and with reliable results.

Here are some of the apps where we used AudioKit:

In Mapping Tonal Harmony Pro and Tessitura Pro we play our own sampled sounds, create and play MIDI sequences and the user is able to mix volume levels, reverb amount and change the tempo LIVE while the sequence is playing

Mapping Tonal Harmony Pro is an app with an interactive map that reveals the secrets of harmony. You can study harmony in different styles from Classical to Jazz and Pop, write your own songs and create charts or use it as a ply along to practice improvisation.

Tessitura Pro contains all scales & modes in music. You can study how modes relate to each other, source scales, tensions vs. chord-tones and also practice using different melodic patterns and approach notes.

In See Music, which is a sight reading app that listens to the player and gives instant note-by-note feedback on their performance, we were able to, not only identify pitch, but also to transcribe the entire performance into standard music notation and include a pitch accuracy report on every note in the score.

When we were designing the app the hardest decision to make was what to use to do pitch recognition.

Implementing code that analyzes audio and turns it into pitch and length information involves lots of advanced math, low level access to memory and functions. Soon we realized this was much tougher than expected. 

After finding AudioKit, we realized that 90% of the work has already been done. The library is simple enough to incorporate to the project, well documented and works really well.

We were able to solve the entire process by just using the AKFrequencyTracker which returns frequency and amplitude.

Since we wanted to analyze an entire musical phrase we needed something a bit more elaborate than a simple tuner.

In our solution we used a timer to stored all the data received from the tracker

      conductor.mic.start()

        conductor.tracker.start()

        timerito = Timer.scheduledTimer(timeInterval: timeBetweenReads,

                                        target: self,

                                        selector: #selector(self.readAndSaveNotes),

                                        userInfo: nil,

                                        repeats: true)

The readAndSaveNotes function simple stores the data at regular intervals (timeBetweenReads) with 3 different listening modes (readStyle)

@objc func readAndSaveNotes() {

        if isListening {

            let amplitude:Float = Float(conductor.tracker.amplitude)

            let frequency:Float = Float(conductor.tracker.frequency)

            if frequency < K.CurrentFreq {

                if (!isRecording && amplitude > minAmpStartTrigger) && (readStyle != K.KReadForASetAmountOfTimeStartRightAway) {

                    isRecording = true

                    listeningStartTime = NSDate().timeIntervalSinceReferenceDate

                }

                if isRecording {

                    switch readStyle {

                    case K.kReadUntilSilence:

                        if amplitude > minAmpEndTrigger {

                            recordNote(f: frequency, a: amplitude)

                        } else if thereIsData {

                            stopListening()

                        }

                        break

                    case K.kReadForASetAmountOfTime:

                        if !isTimeToStop {

                            recordNote(f: frequency, a: amplitude)

                        } else {

                            stopListening(processNotas: true, compareNotas: true)                        

                        }

                        break

                    case K.KReadForASetAmountOfTimeStartRightAway:

                        if !isTimeToStop {

                            recordNote(f: frequency, a: amplitude)

                        } else {

                            stopListening(processNotas: true, compareNotas: true)                        

                        }

                        break

                    case K.kTuning:

                        reportNote(f: frequency, a: amplitude)

                        break

                    default:

                        break

                    }

                    

                }

            }

        }

    }

We found the biggest challenges were: how to ignore background noise and how to re-interpret frequency base on the instrument’s timbre and how to get the starting and ending time of note with accuracy (since the player is playing a melodic line)

Since See Music is an app for all instruments, it must interpret correctly the notes played by instruments with different timbres.

The weight of the overtones is different on every instrument, so the collected frequencies using the AKFrequencyTracker on a single note is usually a set of related frequencies based on the instrument’s timbre.

We found the best way to achieve this was to parametrize the way we collect the data from the AKFrequencyTracker based on each instrument

Here’s an example of the parameters settings for a default instrument:

    var zeroAmplitudThreshold:Float = 0.005

    var noiseAmplitudeThreshold:Float = 0.1  // where notes are probably noise

    var timeBetweenReads:TimeInterval = 0.025  // how fast to read

    var peakThreshold:Float = 0.07 // to consider a new sample a peak

    var minimumNoteDurationInIndexes: Int = 3 // how many samples are good for noteDuration

Also to identify the notes, don’t forget to reduce the frequencies to the pitch and octave that makes sense on an instrument.

Here’s a simple class we used to reduced the frequencies and indentify notes

class MDXSemiFrequencyAmplitude : MDXSimpleFrequencyAmplitude {

    let kpl:MDXPitchListenerConstants = MDXPitchListenerConstants.sharedPitchListenerConstants

    let game:MDXGame = MDXGame.sharedGame

    

    var reducedFrequency:Float = 1.0

    func calcReducedFrequency() {

        var rF:Float = frequency

        let minF:Float = kpl.reducedFreqs[0]

        let maxF:Float = kpl.reducedFreqs[11]

        

        while rF > maxF {

            rF /= 2.0

        }

        while rF < minF {

            rF *= 2.0

        }

        

        reducedFrequency = rF

    }

    

    var expectedRedFreq:Float = 0.0

    

    var expectedFreq:Float {

        get {

            return powf(2, Float(octave)) * expectedRedFreq

        }

    }

    

    var octave:Int = 0

    var midi:Int = 0

    

    func identifyNote() {

        let indexAndWas12:(Index:Int, was12:Bool) = kpl.getNoteIndexByReducedFrequency(reducedFrequency)

        let index = indexAndWas12.Index

        

        if indexAndWas12.was12 {

            reducedFrequency = reducedFrequency / 2

        }

        

        octave = Int(log2f(Float(frequency) / reducedFrequency))

        

        expectedRedFreq = Float(kpl.reducedFreqs[index])

        

        midi = 12 + octave * 12 + index – game.curInstrument.transposition

    }

    

    

    init(_ sfa:MDXSimpleFrequencyAmplitude) {

        super.init(f: sfa.frequency, a: sfa.amplitude)

        tiempo = sfa.tiempo – kpl.listeningStartTime – kpl.timeBetweenReads

        calcReducedFrequency()

        identifyNote()

    }

}

Today I was googling one of my own apps called Mapping Tonal Harmony Pro as I usually do, using the search tools and filtering the results for last-24 hours, to see what new pages were talking about the app or linking to it.

A new version of the app was released three days ago (version 6.5) on the App Store which means there could be no new reviews, for the new version, from users for a couple days (or more.)

Mapping Tonal Harmony Pro has received almost all 5 star reviews and it has an average of 4.5 rating on the App Store

To my surprise, I found the google results for my app on the App Store showing a 1-star Rating!!!

I don’t know how google is obtaining these results, but it is really frustrating. You work so hard on your product to find that a stupid robot routine returns a totally wrong and damaging result.

I, of course, went to the google help forum to start a new topic and see if I could get some answers… I searched for similar topics to see if the issue had been answered and I found an exact topic that was labeled “ANSWERED”! That’s great, I thought, let’s see what they say…

There are hundreds of entries from iOS developers complaining and the answer is:

So the answer’s just a bureaucratic statement with no clear way of knowing when or if the problem will be solved.

It would be so easy and fair to just take the rating away until the problem has been solved. I wonder what the real reason is, if there is one other than negligence.

For now, all developers will have to accept that there apps will show a 1-Star Rating every time the release a new version.

Today we sent the new version 6.3 of Mapping Tonal Harmony Pro to the App Store. This has been one of the biggest projects we’ve ever done. We could have easily created 10 (or more) complete apps with Mapping Tonal Harmony Pro. You can get this incredible app for the price of 2 cups of coffee! but app buyers don’t care, it’s an app, it should be $0.99 or free.

The app works really well, and we are getting great reviews, except when it comes to the infamous USERNAME.

The new version allows the user to use the app without a username (which I strongly recommend implementing in your apps, even if the user can’t access half of the features in them) People hate log-ins, passwords, sign-ups, etc. They are so sick of it, that they don’t even check what information you require them to submit.

In Mapping Tonal Harmony Pro 6.0, we asked the user to create a username and a passcode (so they could access the online database and share progressions with other users.) We didn’t ask for email or any other contact information, we were not trying to collect any data from the users. There was no confirmation process, no facebook, no twitter, nothing at all.
Why did we do this? Because we are also sick of this new data collection trend to target us as a selling point.

Did the users check what they needed to do? NO. As soon as they saw the word USERNAME, that was it! We received many emails asking why we were asking for personal information, even after we showed them there was no personal data required at any moment, they still didn’t like it!

So, lesson learned: “Don’t ask for a username even if you need to” always give the user the option of using the app without a username.

http://mdecks.com

We released Mapping Tonal Harmony Pro 6 a few days ago.

One of the best features in the app is to be able to load and play tracks from the iPad’s Library. The process is pretty straight forward but we’ve encounter a few unexpected problems in the process.

We wanted the user to be able to pick a track from their library and load it into the app.
In order to do this the best option is to use the mediaPicker
This is how you call it:
– (void)chooseASongFromYourLibraryOnView:(UIViewController*) pViewController {
    MPMediaPickerController *picker =
    [[MPMediaPickerController alloc] initWithMediaTypes: MPMediaTypeMusic];
    
    picker.delegate                     = self;
    picker.allowsPickingMultipleItems   = NO;
    picker.prompt                       = NSLocalizedString (@”Select any song from the list”, @”Choose one song only”);
   
    curPickerController=picker;
    [pViewController presentViewController:picker animated:YES completion:nil];
}

Here’s the function called after the user has picked a track
– (void) mediaPicker: (MPMediaPickerController *) mediaPicker didPickMediaItems: (MPMediaItemCollection *) mediaItemCollection
{
    [myPlayer setQueueWithItemCollection:mediaItemCollection];
    NSArray *itemsFromGenericQuery = [mediaItemCollection items];
    self.pickedMediaItemSong=[itemsFromGenericQuery objectAtIndex:0];
    [curPickerController dismissViewControllerAnimated:NO completion:nil];

}

So, we saved the item in pickedMediaItemSong

To play and draw the wave-form representation of the track you need to get the song as a AVURLAsset

asset = [[AVURLAsset alloc] initWithURL:[self.pickedMediaItemSong valueForProperty:MPMediaItemPropertyAssetURL] options:nil];

But, the app would not play items from the cloud

Solution: filter the items shown by the mediapicker to non-cloud items only
add:     picker.showsCloudItems=NO; before you present the ViewController

Now, everything seemed to work fine for a while but every now and then the MPMediaItemPropertyAssetURL would retrun NULL (the asset would end up being nil). And it only happened with a few tracks in our library.

Why? Becuase those tracks had DRM!!

Here’s what Apple Says:

About iTunes Plus

After a few days waiting for review, Sounds of Christmas by mDecks Music is available for sale on the App Store.

mDecks Music Blog’s Post

 

Today, after 13 days waiting for review on the App Store, our new saxophone app “20 Saxophone Tricks of the Trade by Mario Cerra“ changed it status to in Review. Just a few hours later it was processing for the App Store and is now Ready for Sale.

As I’ve explained on the recent post: How to find out the App Store Average Review Time, developers can submit their App Store review waiting time by tweeting using the hashtag: #iosreviewtime

…which I will tweet in a few minutes 🙂

If you are curious about how long the current Average App Store Review Times is, you can go to:
http://appreviewtimes.com/

Also, you can submit your own review time by tweeting as in this example:

60 Top Hat Piano Grooves Vol. 1 by Ariel Ramos took 3 days #iosreviewtime
https://t.co/0hQTehlW3M

— mDecks Music (@mDecksMusic) September 2, 2014

Now that 20 Saxophone Tricks of the Trade is waiting for review, I will add a In App Purchase feature on some of my big apps and make a small version (or free in some cases) of them with limited access and let the user decide if they want to add more features to them by purchasing “modules” within the app.

My current full version is ready for sale on the App Store, and it contains two volumes in one app :

“Rhythmic Vocabulary for All Instruments – Fundamental Studies”

I want to make a Volume 1 (only) version and let the use decide if they want to purchase Vol. 2 within the app

At the moment this is what apple have for a tutorial : https://developer.apple.com/library/ios/technotes/tn2259/_index.html

Here’s my own version on how I did it:

Setting it up

1. I copied the xcode project folder of the full version of the app to another folder.
   Then opened the project with xcode and rename the box Identity and Type and accepted the changes offered by xcode
2. Next I went to the Apple Developer IOS Provisioning website and created a new Identifier for this new version of the app.
   For the Bundle ID I used and Explicit App Id com.mdecks.xxxxx (xxxx being the name I used for the Identity and Type, it doesn’t have to be the same but it is easier this way)
   Make sure the In App purchase option is checked
3. Then created two provisioning profiles for that ID. One Development and one Distribution. Download them and double click the files to add them to xcode
4. Now go to the iTunes Connect website and create the record for the app linking it with the ID I created on the developer site.
   There you can fill as much info as possible. Description, Icon, Screen shots (if you have them already, etc)
5. Then go to the In-App Purchases tab for that app and create a new one. In my case is a non-consumable one since once the user’s paid for it it will be available without expiring.
   There you will have to choose a reference name, and ID, choose the price and add a Language for every language supported by the app (which in my case is two English and Spanish)
   Also you have to select Clear for Sale: YES.  You can upload a screen shot later on, before submitting the app
6. In order to test the In-app purchases in XCode you need to create a sandbox tester user which is done also in iTunes Connect. You can access this on the main page where it says Users and Roles.  I created a couple of them, you can use any emails and names, they do not have to be real ones, just remember the email and password
7. Ok. Back in XCode go to the Build Settings and change the provisioning profile to the one you’ve created for development. If you did it right then in the General tab the Identity shouldn’t show any problems.
8. Enable In-App Purchase in the Capabilities tab

Implementing it

1. There’s an amazing tutorial by Ray Wenderlich with a sample code download. I suggest you read it since is very well put together.
2. In Ray’s solution there’s a class that takes care of all the In-App purchase methods and events. Then you create a subclass of that one and tweak it for your products and needs, and a TableViewController subclass that will show all the In-App purchases IDs you’ve created for your app, letting the user buy any of them from that table. There’s also a restore button to retrieve the in-app purchase IDs that have been purchased by the current user (very useful when the user has deleted the app and reinstalled it)
3. Note 1: One of the things I was confused about was the productIdentifiers. When I tried the first time I got an  in-app purchase invalid product. I finally got it working by just using the ID for each in-App purchase as shown in iTunes, without any prefixes (just the productID that’s it !)
4. Note 2: The simulator in XCode did retrieve the list of products without a problem, but to try the entire purchase process you must run your app in a real device, (previously signing out of the App Store) and then at the moment of purchasing use one of your SandBox Tester Users (and it will show as such in the alert dialogue on your device)

That’s all. Now the app is waiting for review on the app store

“Rhythmic Vocabulary for All Instruments – Fundamental Studies”

I also added some in-app purchases to the guitar app which is also waiting for review on the app store:

 “60 Top Hat Guitar Licks Vol. 1”

And I am working on adding 4 modules of in-app purchases to the free version of my piano app:

 “60 Top Hat Guitar Licks Vol. 1”