As you try new service, one is bound to make a lot of mess. I did the same with my Team Foundation Services account. After initial testing was done I decided to make a clean start and delete a project.
It took me quite a while to give up on finding delete. It seems that somebody forgot to take his pills during design phase and thus delete button does not exist.
Fortunately, this is just an Team Foundation Server so good old command line will do nicely:
CD "C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE"
TFSDeleteProject.exe /collection:https://jmedved.visualstudio.com/DefaultCollection/ Test
Warning: Deleting a team project is an irrecoverable operation. All version control, work item tracking and Team Foundation build data will be destroyed from the system. The only way to recover this data is by restoring a stored backup of the databases. Are you sure you want to delete the team project and all of its data (Y/N)? Y
Deleting from Build ...
Done
Deleting from Version Control ...
Done
Deleting from Work Item Tracking ...
Done
Deleting from TestManagement ...
Done
Deleting from ProcessManagement ...
Done
Deleting from LabManagement ...
Done
Deleting from ProjectServer ...
Done
Warning. Did not find Report Server service.
Warning. Did not find SharePoint site service.
Deleting from Team Foundation Core ...
Done
Of course, if you want to create new project with same name, do not forget to visit File
-> Source Control
-> Advanced
-> Workspaces
in order to remove last traces.