Currently, if you have two windows in the default activity, and some more in a second activity, the first two shortcuts always bring you to the default/first activity.
I think it would make more sense if they would take you to the windows as seen on the task bar, which is distinct in every activity. Is this a bug worth reporting, or is it intended to be this way?