Halo 4 Stats - with API (Finally)

[Xerax]

So for the past month I've been working with a friend to look into the Halo 4 Stats website on http://app.halowaypoint.com/. And finally, after a lot of work, we're ready to release the Source Code, how to access and use the internal API and a small example website to show off basics of what can be done.

Developers

- We have a wiki of information on how to Authenticate Endpoints, and send requests

- We have examples on how to use the API

Users

- We have a couple of examples that are ready for users to use

- http://auntiedot.net/ - http://leafapp.co/

For more information, visit: http://auntiedot.net

(Also, if you could upvote it on reddit (http://www.reddit.com/r/halo/comments/17qkz5/halo_4_stats_with_api_finally/) that would be awesome, so that the Halo Community can start using it.

[Guest]

An internal API? Interesting.. nice discover/SDK. :D

[iLoch]

Really? You're going to take credit for this? I, and many others, have had access to this API since it launched on November 4th. We've specifically avoided releasing the information to the public because people do not understand how to use APIs. Now - thanks to your efforts - the API will be restricted as it get's slammed by sloppy programmers with useless applications. I've been working with HaloTracker in order to get stats working on that site (which has over 100,000 registered gamertags) and now they're at risk of being blocked because of other people abusing the API.

Please be more responsible next time you decide to release a private API. Do so from a closed environment. Provide developers with access through a proxy, not direct access. Say goodbye to the API.

[Xerax]

Really? You're going to take credit for this? I, and many others, have had access to this API since it launched on November 4th. We've specifically avoided releasing the information to the public because people do not understand how to use APIs. Now - thanks to your efforts - the API will be restricted as it get's slammed by sloppy programmers with useless applications. I've been working with HaloTracker in order to get stats working on that site (which has over 100,000 registered gamertags) and now they're at risk of being blocked because of other people abusing the API.

Please be more responsible next time you decide to release a private API. Do so from a closed environment. Provide developers with access through a proxy, not direct access. Say goodbye to the API.

Sorry, did we knock you from your Ivory Tower? Some people believe sharing is caring. And lets face it, 343 know people have been doing this. If they cared, they would of taken action already.

[iLoch]

Sorry, did we knock you from your Ivory Tower? Some people believe sharing is caring. And lets face it, 343 know people have been doing this. If they cared, they would of taken action already.

The most responsible developers in the community have had access for a while. We know sharing is caring. There's a right, and a wrong way of doing it. (See http://halocharts.com JSON service) By releasing this source code, you've allowed anyone to hit the service with a million requests. 343i already have tightened the API - my guess is that you didn't notice since you didn't get access right away. Next step is an IP whitelist.

[Xerax]

The most responsible developers in the community have had access for a while. We know sharing is caring. There's a right, and a wrong way of doing it. (See http://halocharts.com JSON service) By releasing this source code, you've allowed anyone to hit the service with a million requests. 343i already have tightened the API - my guess is that you didn't notice since you didn't get access right away. Next step is an IP whitelist.

IP Whitelist? Please. Stop.

[Hardcore Til I Die]

Really? You're going to take credit for this? I, and many others, have had access to this API since it launched on November 4th. We've specifically avoided releasing the information to the public because people do not understand how to use APIs. Now - thanks to your efforts - the API will be restricted as it get's slammed by sloppy programmers with useless applications. I've been working with HaloTracker in order to get stats working on that site (which has over 100,000 registered gamertags) and now they're at risk of being blocked because of other people abusing the API.

Please be more responsible next time you decide to release a private API. Do so from a closed environment. Provide developers with access through a proxy, not direct access. Say goodbye to the API.

With all due respect they MUST have expected this to happen.

They can't release an API to a bunch of people outside their organisation and expect it not to be passed around.

[iLoch]

IP Whitelist? Please. Stop.

Sorry, is that hard for you to believe? That's actually just one line they would have to add to block 100% of the unwanted access. Are you kidding me?

[BillinghamJ]

FYI - I'm the other developer/researcher of this stuff.

The most responsible developers in the community have had access for a while. We know sharing is caring. There's a right, and a wrong way of doing it. (See http://halocharts.com JSON service) By releasing this source code, you've allowed anyone to hit the service with a million requests. 343i already have tightened the API - my guess is that you didn't notice since you didn't get access right away. Next step is an IP whitelist.

Do you actually know what you're talking about?

It is a JSON API, which is accessed by the HWP website through AJAX. Each user makes their own requests to it - therefore IP whitelisting would be impossible.

The security on it is extremely basic at present, so I'm not sure what you mean by "343i already have tightened the API" since it's really not in the slightest bit difficult to authenticate to it.

An intermediate service (rather than documenting how to access directly) would be an incredibly stupid thing to do for several reasons:

• It would be painfully obvious since a ridiculous number of requests to different accounts would be coming from the same IP
  
• It would be incredibly easy to block
  
• It would require working around, rather than with, the authentication system. Our docs & client conform fully to the way the auth system was intended to work and, as such, there are no security issues in terms of phishing & no reason for 343/MS to get ****ed off about it
  

If anyone does use our library to send millions of requests or make an intermediate service, they're an idiot, but luckily it's easy for 343 to block. The research/library should only be used (& will only work long term) as part of a client application executing on the user's computer - whether that be with JavaScript or a compiled app.

Additionally, this is not an internal API. A private one, maybe, but every time you visit HWP you directly call it 20-100 times (ish), so it's not exactly hidden.

Sorry, is that hard for you to believe? That's actually just one line they would have to add to block 100% of the unwanted access. Are you kidding me?

No, no. Are you kidding me? **** off and come back when you actually know what AJAX is and how you build a Javascript based client side web app.

[iLoch]

With all due respect they MUST have expected this to happen.

They can't release an API to a bunch of people outside their organisation and expect it not to be passed around.

Well they're using the best authentication they can get - so my guess is that they're trying to provide their users with the best possible experience at the expense of data security. Now they're just going to clamp down on it. It's nothing to them. Of course they expected it to happen - but they weren't going to do anything about it if it was just a few community sites accessing the information. Now this is a real problem, and they'll find a solution.

[BillinghamJ]

Well they're using the best authentication they can get - so my guess is that they're trying to provide their users with the best possible experience at the expense of data security. Now they're just going to clamp down on it. It's nothing to them. Of course they expected it to happen - but they weren't going to do anything about it if it was just a few community sites accessing the information. Now this is a real problem, and they'll find a solution.

Tbh, I don't think they're bothered. The programme manager of the Halo web team followed Xerax & I on Twitter. I messaged him letting him know we're more than happy to work with them if they don't like what we're doing or want us to do something for them.

[iLoch]

FYI - I'm the other developer/researcher of this stuff.

Do you actually know what you're talking about?

It is a JSON API, which is accessed by the HWP website through AJAX. Each user makes their own requests to it - therefore IP whitelisting would be impossible.

The security on it is extremely basic at present, so I'm not sure what you mean by "343i already have tightened the API" since it's really not in the slightest bit difficult to authenticate to it.

An intermediate service (rather than documenting how to access directly) would be an incredibly stupid thing to do for several reasons:

• It would be painfully obvious since a ridiculous number of requests to different accounts would be coming from the same IP
  
• It would be incredibly easy to block
  
• It would require working around, rather than with, the authentication system. Our docs & client conform fully to the way the auth system was intended to work and, as such, there are no security issues in terms of phishing & no reason for 343/MS to get ****ed off about it
  

If anyone does use our library to send millions of requests or make an intermediate service, they're an idiot, but luckily it's easy for 343 to block. The research/library should only be used (& will only work long term) as part of a client application executing on the user's computer - whether that be with JavaScript or a compiled app.

Additionally, this is not an internal API. A private one, maybe, but every time you visit HWP you directly call it 20-100 times (ish), so it's not exactly hidden.

No, no. Are you kidding me? **** off and come back when you actually know what AJAX is and how you build a Javascript based client side web app.

Alright, so I guess the fact that I have been doing this for seven years doesn't mean anything. Apparently you don't understand AJAX. You cannot make requests to the API clientside from your users' computer (through a web browser), because of cross-origin access policies. You are forced to send connections through your server, which means it's one IP to block all requests you want to make from your site. What you're describing, and the API you provided, works great for desktop applications, but will completely **** over websites like HaloTracker, which is where the majority of the calls are coming from, not desktop applications.

And as I've already stated, I had access to this API the same day it was released and was able to fully authenticate without problems. When it was first released, they did not require any data-scraping, it was 100% header based. They've locked it down more by embedding it within the page.

[BillinghamJ]

Alright, so I guess the fact that I have been doing this for seven years doesn't mean anything. Apparently you don't understand AJAX. You cannot make requests to the API clientside from your users' computer, because of cross-origin access policies. You are forced to send connections through your server, which means it's one IP to block all requests you want to make from your site. What you're describing, and the API you provided, works great for desktop applications, but will completely **** over websites like HaloTracker, which is where the majority of the calls are coming from, not desktop applications.

And as I've already stated, I had access to this API the same day it was released and was able to fully authenticate without problems. When it was first released, they did not require any data-scraping, it was 100% header based. They've locked it down more by embedding it within the page.

'Tis a good point which I hadn't considered, but regardless, going through a server is never going to be a sustainable way of accessing an API like this.

Also, I've been doing this for more than seven years, so ;)

[iLoch]

'Tis a good point which I hadn't considered, but regardless, going through a server is never going to be a sustainable way of accessing an API like this.

Also, I've been doing this for more than seven years, so ;)

Not long enough then if you hadn't considered that, so... ;) And for most uses, that's the only way of accessing the API, which is why everyone with a big site is extremely ticked off at you two right now.

[BillinghamJ]

Not long enough then if you hadn't considered that, so... ;) And for most uses, that's the only way of accessing the API, which is why everyone with a big site is extremely ticked off at you two right now.

To be fair, anyone with a big site wouldn't have lasted long anyway. Us doing this is unlikely to make much difference.

[iLoch]

To be fair, anyone with a big site wouldn't have lasted long anyway. Us doing this is unlikely to make much difference.

Like I said, HaloTracker is running thousands of queries daily. My guess is that 343i is fully aware and they are turning a blind eye. You've put HTR in the spotlight and now 343i will be forced to act because of all the extra incoming requests - and they'll be blocking any and all outside websites sending a sufficient amount of requests. Next time provide an unauthenticated throttled JSON service like Firestream has, and save the authenticated requests for Microsoft, 3rd parties, and people who have figured it out on their own.

[iBotPeaches]

The most responsible developers in the community have had access for a while. We know sharing is caring. There's a right, and a wrong way of doing it. (See http://halocharts.com JSON service) By releasing this source code, you've allowed anyone to hit the service with a million requests. 343i already have tightened the API - my guess is that you didn't notice since you didn't get access right away. Next step is an IP whitelist.

Unless I'm an idiot. Which is entirely possible. That "public" halocharts JSON has 2 endpoints. Which sure left a lot of room for creativity -_-

No one here is trying to make another HaloTracker. The only tone I'm getting from you, is that you want to remain the only stat tracking website. The problem is though, everyone likes their stats differently. I don't want an over-bloated and what I think poor design. I want stats in a quick and modern design, which is why I started Leaf. HaloWaypoint is too flashy for my consumption, and unless halo.junk.ws made a H4 version I have no where to go.

I wonder, were you one of the developers who were invited to the private Bungie API as it was made? I don't recognize your username from the 20ish of us that were there. That group of developers shared knowledge and we all worked together to create better applications for all. There was a HaloTracker dev in that group, but I don't think it was you. Point being, we were all friends working for the end user. Which you are forgetting. So as you continue to pull your "special responsible" developer card, I will simply work on Leaf.

<3 peaches

[iLoch]

I'm not sure which group you are referring to, I was not apart of that. I'm also not a developer for HaloTracker, but I did help xorth get the site working with regards to the Halo 4 API - which is why HTR never skipped a beat. I'm going to assume the Reach API group you're referring to was created after the announcement of the closing of Bungie's API. My response is that I had access to 343i's API about a month prior to that all happening so I would have had no use for that group. My position is the same as everyone else's in here: open stats for everybody. There's a reason they aren't though, and that's because there are plenty of people with ill intent that will use the service to do harm (namely just slam it with requests) and so I've never released anything to the public. It only makes sense.

[Xerax]

I'm not sure which group you are referring to, I was not apart of that. I'm also not a developer for HaloTracker, but I did help xorth get the site working with regards to the Halo 4 API - which is why HTR never skipped a beat. I'm going to assume the Reach API group you're referring to was created after the announcement of the closing of Bungie's API. My response is that I had access to 343i's API about a month prior to that all happening so I would have had no use for that group. My position is the same as everyone else's in here: open stats for everybody. There's a reason they aren't though, and that's because there are plenty of people with ill intent that will use the service to do harm (namely just slam it with requests) and so I've never released anything to the public. It only makes sense.

Some people will do bad things, so punish everyone who won't.

This is why people can't have nice things.

[iLoch]

Yeah you're exactly right. We shouldn't give people nice things if they're going to abuse them. The only people who deserve to have access to this API at this point are the people who figured it out for themselves.