Category: Uncategorized

mDecks Music Apps That Use AudioKit

At mDecks Music we develop apps to study and practice music and music theory, mostly for intermediate to advanced musicians.

youtubeface2.jpg

Our apps need to interact with the user in many ways: from playing simple chords using sampled sounds to complex accompaniments, received and send MIDI data and also listen to audio input from the user and convert into useful musical information.

Without a doubt, using AudioKit allowed us to achieve all these tasks easily and with reliable results.

Here are some of the apps where we used AudioKit:

In Mapping Tonal Harmony Pro and Tessitura Pro we play our own sampled sounds, create and play MIDI sequences and the user is able to mix volume levels, reverb amount and change the tempo LIVE while the sequence is playing

Mapping Tonal Harmony Pro is an app with an interactive map that reveals the secrets of harmony. You can study harmony in different styles from Classical to Jazz and Pop, write your own songs and create charts or use it as a ply along to practice improvisation.

mthpro6prom1.jpg

Tessitura Pro contains all scales & modes in music. You can study how modes relate to each other, source scales, tensions vs. chord-tones and also practice using different melodic patterns and approach notes.

In See Music, which is a sight reading app that listens to the player and gives instant note-by-note feedback on their performance, we were able to, not only identify pitch, but also to transcribe the entire performance into standard music notation and include a pitch accuracy report on every note in the score.see-music.jpg

When we were designing the app the hardest decision to make was what to use to do pitch recognition.

Implementing code that analyzes audio and turns it into pitch and length information involves lots of advanced math, low level access to memory and functions. Soon we realized this was much tougher than expected. 

After finding AudioKit, we realized that 90% of the work has already been done. The library is simple enough to incorporate to the project, well documented and works really well.

We were able to solve the entire process by just using the AKFrequencyTracker which returns frequency and amplitude.

Since we wanted to analyze an entire musical phrase we needed something a bit more elaborate than a simple tuner.

In our solution we used a timer to stored all the data received from the tracker

      conductor.mic.start()

        conductor.tracker.start()

        timerito = Timer.scheduledTimer(timeInterval: timeBetweenReads,

                                        target: self,

                                        selector: #selector(self.readAndSaveNotes),

                                        userInfo: nil,

                                        repeats: true)

The readAndSaveNotes function simple stores the data at regular intervals (timeBetweenReads) with 3 different listening modes (readStyle)

@objc func readAndSaveNotes() {

        if isListening {

            let amplitude:Float = Float(conductor.tracker.amplitude)

            let frequency:Float = Float(conductor.tracker.frequency)

            if frequency < K.CurrentFreq {

                if (!isRecording && amplitude > minAmpStartTrigger) && (readStyle != K.KReadForASetAmountOfTimeStartRightAway) {

                    isRecording = true

                    listeningStartTime = NSDate().timeIntervalSinceReferenceDate

                }

                if isRecording {

                    switch readStyle {

                    case K.kReadUntilSilence:

                        if amplitude > minAmpEndTrigger {

                            recordNote(f: frequency, a: amplitude)

                        } else if thereIsData {

                            stopListening()

                        }

                        break

                    case K.kReadForASetAmountOfTime:

                        if !isTimeToStop {

                            recordNote(f: frequency, a: amplitude)

                        } else {

                            stopListening(processNotas: true, compareNotas: true)                        

                        }

                        break

                    case K.KReadForASetAmountOfTimeStartRightAway:

                        if !isTimeToStop {

                            recordNote(f: frequency, a: amplitude)

                        } else {

                            stopListening(processNotas: true, compareNotas: true)                        

                        }

                        break

                    case K.kTuning:

                        reportNote(f: frequency, a: amplitude)

                        break

                    default:

                        break

                    }

                    

                }

            }

        }

    }

We found the biggest challenges were: how to ignore background noise and how to re-interpret frequency base on the instrument’s timbre and how to get the starting and ending time of note with accuracy (since the player is playing a melodic line)

Since See Music is an app for all instruments, it must interpret correctly the notes played by instruments with different timbres.

The weight of the overtones is different on every instrument, so the collected frequencies using the AKFrequencyTracker on a single note is usually a set of related frequencies based on the instrument’s timbre.

We found the best way to achieve this was to parametrize the way we collect the data from the AKFrequencyTracker based on each instrument

Here’s an example of the parameters settings for a default instrument:

    var zeroAmplitudThreshold:Float = 0.005

    var noiseAmplitudeThreshold:Float = 0.1  // where notes are probably noise

    var timeBetweenReads:TimeInterval = 0.025  // how fast to read

    var peakThreshold:Float = 0.07 // to consider a new sample a peak

    var minimumNoteDurationInIndexes: Int = 3 // how many samples are good for noteDuration

Also to identify the notes, don’t forget to reduce the frequencies to the pitch and octave that makes sense on an instrument.

Here’s a simple class we used to reduced the frequencies and indentify notes

class MDXSemiFrequencyAmplitude : MDXSimpleFrequencyAmplitude {

    let kpl:MDXPitchListenerConstants = MDXPitchListenerConstants.sharedPitchListenerConstants

    let game:MDXGame = MDXGame.sharedGame

    

    var reducedFrequency:Float = 1.0

    func calcReducedFrequency() {

        var rF:Float = frequency

        let minF:Float = kpl.reducedFreqs[0]

        let maxF:Float = kpl.reducedFreqs[11]

        

        while rF > maxF {

            rF /= 2.0

        }

        while rF < minF {

            rF *= 2.0

        }

        

        reducedFrequency = rF

    }

    

    var expectedRedFreq:Float = 0.0

    

    var expectedFreq:Float {

        get {

            return powf(2, Float(octave)) * expectedRedFreq

        }

    }

    

    var octave:Int = 0

    var midi:Int = 0

    

    func identifyNote() {

        let indexAndWas12:(Index:Int, was12:Bool) = kpl.getNoteIndexByReducedFrequency(reducedFrequency)

        let index = indexAndWas12.Index

        

        if indexAndWas12.was12 {

            reducedFrequency = reducedFrequency / 2

        }

        

        octave = Int(log2f(Float(frequency) / reducedFrequency))

        

        expectedRedFreq = Float(kpl.reducedFreqs[index])

        

        midi = 12 + octave * 12 + index – game.curInstrument.transposition

    }

    

    

    init(_ sfa:MDXSimpleFrequencyAmplitude) {

        super.init(f: sfa.frequency, a: sfa.amplitude)

        tiempo = sfa.tiempo – kpl.listeningStartTime – kpl.timeBetweenReads

        calcReducedFrequency()

        identifyNote()

    }

}

Advertisements

Google Rating nightmare for iOS Apps

Today I was googling one of my own apps called Mapping Tonal Harmony Pro as I usually do, using the search tools and filtering the results for last-24 hours, to see what new pages were talking about the app or linking to it.

A new version of the app was released three days ago (version 6.5) on the App Store which means there could be no new reviews, for the new version, from users for a couple days (or more.)

Mapping Tonal Harmony Pro has received almost all 5 star reviews and it has an average of 4.5 rating on the App Store

Screen Shot 2015-11-30 at 9.44.41 PM.png

To my surprise, I found the google results for my app on the App Store showing a 1-star Rating!!!

Screen Shot 2015-11-30 at 9.40.07 PM.png

I don’t know how google is obtaining these results, but it is really frustrating. You work so hard on your product to find that a stupid robot routine returns a totally wrong and damaging result.

I, of course, went to the google help forum to start a new topic and see if I could get some answers… I searched for similar topics to see if the issue had been answered and I found an exact topic that was labeled “ANSWERED”! That’s great, I thought, let’s see what they say…

There are hundreds of entries from iOS developers complaining and the answer is:

Screen Shot 2015-11-30 at 10.15.07 PM

So the answer’s just a bureaucratic statement with no clear way of knowing when or if the problem will be solved.

It would be so easy and fair to just take the rating away until the problem has been solved. I wonder what the real reason is, if there is one other than negligence.

For now, all developers will have to accept that there apps will show a 1-Star Rating every time the release a new version.

The negative power of usernames and how to solve it

Mapping Tonal Harmony Pro 6

Today we sent the new version 6.3 of Mapping Tonal Harmony Pro to the App Store. This has been one of the biggest projects we’ve ever done. We could have easily created 10 (or more) complete apps with Mapping Tonal Harmony Pro. You can get this incredible app for the price of 2 cups of coffee! but app buyers don’t care, it’s an app, it should be $0.99 or free.

The app works really well, and we are getting great reviews, except when it comes to the infamous USERNAME.

The new version allows the user to use the app without a username (which I strongly recommend implementing in your apps, even if the user can’t access half of the features in them) People hate log-ins, passwords, sign-ups, etc. They are so sick of it, that they don’t even check what information you require them to submit.

In Mapping Tonal Harmony Pro 6.0, we asked the user to create a username and a passcode (so they could access the online database and share progressions with other users.) We didn’t ask for email or any other contact information, we were not trying to collect any data from the users. There was no confirmation process, no facebook, no twitter, nothing at all.
Why did we do this? Because we are also sick of this new data collection trend to target us as a selling point.

Did the users check what they needed to do? NO. As soon as they saw the word USERNAME, that was it! We received many emails asking why we were asking for personal information, even after we showed them there was no personal data required at any moment, they still didn’t like it!

So, lesson learned: “Don’t ask for a username even if you need to” always give the user the option of using the app without a username.

http://mdecks.com

Playing music within your app

We released Mapping Tonal Harmony Pro 6 a few days ago.

One of the best features in the app is to be able to load and play tracks from the iPad’s Library. The process is pretty straight forward but we’ve encounter a few unexpected problems in the process.

We wanted the user to be able to pick a track from their library and load it into the app.
In order to do this the best option is to use the mediaPicker
This is how you call it:
– (void)chooseASongFromYourLibraryOnView:(UIViewController*) pViewController {
    MPMediaPickerController *picker =
    [[MPMediaPickerController alloc] initWithMediaTypes: MPMediaTypeMusic];
    
    picker.delegate                     = self;
    picker.allowsPickingMultipleItems   = NO;
    picker.prompt                       = NSLocalizedString (@”Select any song from the list”, @”Choose one song only”);
   
    curPickerController=picker;
    [pViewController presentViewController:picker animated:YES completion:nil];
}

Here’s the function called after the user has picked a track
– (void) mediaPicker: (MPMediaPickerController *) mediaPicker didPickMediaItems: (MPMediaItemCollection *) mediaItemCollection
{
    [myPlayer setQueueWithItemCollection:mediaItemCollection];
    NSArray *itemsFromGenericQuery = [mediaItemCollection items];
    self.pickedMediaItemSong=[itemsFromGenericQuery objectAtIndex:0];
    [curPickerController dismissViewControllerAnimated:NO completion:nil];

}

So, we saved the item in pickedMediaItemSong

To play and draw the wave-form representation of the track you need to get the song as a AVURLAsset

asset = [[AVURLAsset alloc] initWithURL:[self.pickedMediaItemSong valueForProperty:MPMediaItemPropertyAssetURL] options:nil];

But, the app would not play items from the cloud

Solution: filter the items shown by the mediapicker to non-cloud items only
add:     picker.showsCloudItems=NO; before you present the ViewController

Now, everything seemed to work fine for a while but every now and then the MPMediaItemPropertyAssetURL would retrun NULL (the asset would end up being nil). And it only happened with a few tracks in our library.

Why? Becuase those tracks had DRM!!

Here’s what Apple Says:

About iTunes Plus

Learn more about iTunes Plus, the high-quality format of songs and music videos available through the iTunes Store.

iTunes Plus refers to songs and music videos in high quality AAC format that don’t have Digital Rights Management (DRM).

All songs and music videos now for sale in the iTunes Store are iTunes Plus. In some cases, if you previously bought music with DRM from the iTunes Store, you can download the higher quality, DRM-free versions of your songs with a subscription to iTunes Match. The tracks must show as Matched or Purchased in the iCloud Status column in your iTunes library, and the same album or song must still be available in the iTunes Store.

To upgrade your music to iTunes Plus, follow these steps on your computer:

  1. Open iTunes.
  2. If you’re not already, sign in with your Apple ID and password.
  3. Click the My Music tab at the top of iTunes.
  4. Click the song or album you want to upgrade.
  5. Press the delete key on your keyboard. In the message that appears, click Move to Trash.
  6. Click the iTunes Store tab at the top of iTunes.
  7. Under Quick Links on the right-hand side of iTunes, click Purchased.
  8. Click Music in the upper-right corner of iTunes.
  9. Find the song or album you want to upgrade.
  10. Click the iCloud Download icon   on the song or album to download the new version.

Building an App on iOS 8 still compatible with iOS7

My new app 20 Saxophone Tricks of the Trade by Mario Cerra has been available on the App Store for over a week now.

We’ve had many downloads apparently with no issues. Yesterday a user wrote us an email saying the app was crashing right after launching. I knew right away it had something to do with the user’s device or IOS.

I downloaded the app from the App Store on an iPad running IOS 7.1.1 and the app crashed right away. I tried it in XCode with the iPad 2 simulator and the app run perfectly but, as soon I connected my iPad with IOS 7 and run it from XCode the app crashed again with an uncaught exemption error.

Long story short, after a few break points I found the issue was in a line where I’ve used containsString to find if a substring is contained in a string, which is an iOS 8 only method. Although the app was set to build for ios 7.1 and up the xcode compiler gave me no warning notice.

The solution is to use the rangeOfString method that is compatible with iOS 7:

[yourString rangeOfString:@”whatever-you-are-looking-for string”].location == NSNotFound

That solved the problem.
To submit a new version of the app to the App Store you must change the version number and build number of the app, archive it and submit it.

Now the app is back on waiting for review on the App Store.

I also added a note in the description for the current version of the app so users still running IOS 7 will wait for the update before buying it.

 

– (NSString*) archivoFullPath: (NSString *)archivo {
NSMutableString* result = [NSMutableString stringWithString: mDecksURLFolder];
if (!([archivo rangeOfString:@”http”].location == NSNotFound)) {
return archivo;
} else {
[result appendString: archivo];
return result;
}
}

20 Saxophone Tricks of the Trade by Mario Cerra is ready for sale

saxiconToday, after 13 days waiting for review on the App Store, our new saxophone app 20 Saxophone Tricks of the Trade by Mario Cerra changed it status to in Review. Just a few hours later it was processing for the App Store and is now Ready for Sale.

As I’ve explained on the recent post: How to find out the App Store Average Review Time, developers can submit their App Store review waiting time by tweeting using the hashtag: #iosreviewtime

…which I will tweet in a few minutes 🙂

In App Purchase

Now that 20 Saxophone Tricks of the Trade is waiting for review, I will add a In App Purchase feature on some of my big apps and make a small version (or free in some cases) of them with limited access and let the user decide if they want to add more features to them by purchasing “modules” within the app.

My current full version is ready for sale on the App Store, and it contains two volumes in one app :

rhythm app “Rhythmic Vocabulary for All Instruments – Fundamental Studies”

I want to make a Volume 1 (only) version and let the use decide if they want to purchase Vol. 2 within the app

At the moment this is what apple have for a tutorial : https://developer.apple.com/library/ios/technotes/tn2259/_index.html

Here’s my own version on how I did it:

Setting it up

  1. I copied the xcode project folder of the full version of the app to another folder.
    Then opened the project with xcode and rename the box Identity and Type and accepted the changes offered by xcode
  2. Next I went to the Apple Developer IOS Provisioning website and created a new Identifier for this new version of the app.
    For the Bundle ID I used and Explicit App Id com.mdecks.xxxxx (xxxx being the name I used for the Identity and Type, it doesn’t have to be the same but it is easier this way)
    Make sure the In App purchase option is checked
  3. Then created two provisioning profiles for that ID. One Development and one Distribution. Download them and double click the files to add them to xcode
  4. Now go to the iTunes Connect website and create the record for the app linking it with the ID I created on the developer site.
    There you can fill as much info as possible. Description, Icon, Screen shots (if you have them already, etc)
  5. Then go to the In-App Purchases tab for that app and create a new one. In my case is a non-consumable one since once the user’s paid for it it will be available without expiring.
    There you will have to choose a reference name, and ID, choose the price and add a Language for every language supported by the app (which in my case is two English and Spanish)
    Also you have to select Clear for Sale: YES.  You can upload a screen shot later on, before submitting the app
  6. In order to test the In-app purchases in XCode you need to create a sandbox tester user which is done also in iTunes Connect. You can access this on the main page where it says Users and Roles.  I created a couple of them, you can use any emails and names, they do not have to be real ones, just remember the email and password
  7. Ok. Back in XCode go to the Build Settings and change the provisioning profile to the one you’ve created for development. If you did it right then in the General tab the Identity shouldn’t show any problems.
  8. Enable In-App Purchase in the Capabilities tab

Implementing it

  1. There’s an amazing tutorial by Ray Wenderlich with a sample code download. I suggest you read it since is very well put together.
  2. In Ray’s solution there’s a class that takes care of all the In-App purchase methods and events. Then you create a subclass of that one and tweak it for your products and needs, and a TableViewController subclass that will show all the In-App purchases IDs you’ve created for your app, letting the user buy any of them from that table. There’s also a restore button to retrieve the in-app purchase IDs that have been purchased by the current user (very useful when the user has deleted the app and reinstalled it)
  3. Note 1: One of the things I was confused about was the productIdentifiers. When I tried the first time I got an  in-app purchase invalid product. I finally got it working by just using the ID for each in-App purchase as shown in iTunes, without any prefixes (just the productID that’s it !)
  4. Note 2: The simulator in XCode did retrieve the list of products without a problem, but to try the entire purchase process you must run your app in a real device, (previously signing out of the App Store) and then at the moment of purchasing use one of your SandBox Tester Users (and it will show as such in the alert dialogue on your device)

That’s all. Now the app is waiting for review on the app store

rhythm app “Rhythmic Vocabulary for All Instruments – Fundamental Studies”

I also added some in-app purchases to the guitar app which is also waiting for review on the app store:

guitar app “60 Top Hat Guitar Licks Vol. 1”

And I am working on adding 4 modules of in-app purchases to the free version of my piano app:

piano app “60 Top Hat Guitar Licks Vol. 1”