Astro Empires Database

This is where the best suggestions are moved to, so discussion can carry on with moderation and be more easily read by the volunteers and development team.

Moderator: Support Moderators

Forum rules
Opening new topics in this forum is not possible, you may only reply to existing topics.

Only users with 50 or more posts can reply to topics.

This forum is moderated, so any posts will have to be approved by a moderator before being published.
User avatar
Winchester
Addicted Member
Addicted Member
Posts: 17730
Joined: Tue 24 Mar, 2009 00:44
Reputation: 766
Location: The World Wide Cesspool

Astro Empires Database

Postby Winchester » Tue 09 Nov, 2010 23:21

Preliminaries:

First, I would like to thank everyone that participated in the concept discussion on the topic; your help has been appreciated. Second, I would like to state that I will be making this as concise as possible; however, it is a very in-depth request, and thus will inevitably contain various walls of text. If you are unwilling to read the first post, do us all a favor, and don’t post. Third, I would like to point out that I have a new take on this concept, and thus do not violate the FMR; likewise, I have direct approval from Wizard to make this thread, so keep that in mind if you want to accuse me of breaking the rules.

Now then. The request here is for a database/parser, for use by the players. There are three rough sides to the argument: the side that says we shouldn’t add one, the side that says we should limit it to prevent it from being overpowered, and the side that says we need to make it as useful as the current ones, or people will keep using the illegal databases/parsers. While the first opinion is one that will necessarily be put out by this very suggestion, I hope to tread the middle way between the latter two; we will see how successful I am in the following paragraphs.

The reason for adding an AE-run database is simple: the players want it, and people are already using illegal scripts to achieve the same effect. By adding one, we improve the game, make the players happy, and cut down on the people that need to cheat to remain competitive. There was some heated debate over whether to make this free, usable with the normal upgrade, or of making an “elite” upgrade. Well, again, I hope to take the middle way; this thread will coincide with another one that I am making, which will propose a new form of elite upgrade.

Without further ado, onto the idea. I will be approaching it topically, so scan the bolded/underlined headers to see which section your questions might be in.

What information is held?

The DB will contain information on individual astros, under the following format:

Coordinates
Astro type
Base owner name
Base econ
Base jumpgate
# of levels for the two highest defenses
# of command centers
# of fleets (per guild) (landed and incoming)
Total credit value of fleets (per guild) (landed and incoming)
Fleet owner’s name/fleet size (only for the five largest fleets) (landed only)

Example:

XX:XX:XX:XX – [SEXY] Ribbentrop (666/666) JG: 20 PS: 50 PR: 100 CC: 30
[SEXY] 69 fleets – 1,337,000,000 total fleet
[UGLY] 1 fleet – 40 total fleet
[SEXY] Ribbentrop 337,000,000
[SEXY] Ikari 100,000,000
[SEXY] The Dude 100,000,000
[SEXY] Ferdoc 100,000,000
[SEXY] tallwhiteninja 100,000,000

The reason for this sort of info is, in my opinion, simple; the database that I used to use, way back in the day, gave you coordinates, base owner names, JG/defense levels, and info on every fleet there. I’m treading a middle ground by having the DB show number of fleets/total credit value of fleets per guild, with the five largest fleets visible. This way, you don’t have access to every individual fleet on the blob, but you can still see where the blob is, and how big it is. Likewise, by having the five largest fleets seen on the DB, you can check for large fleets sitting off-base.

Note that any coordinates without a player name are uncolonized; this means that the DB can be used to find open astros to colonize, as it tracks astro type.

Who can see this info?

While it has been suggested that this be a guild-specific thing (and others suggested it be player-only), I’ve once again found the middle ground.

1: Anything that you personally scan will always be accessible to you (barring information decay, which we will discuss later). If you leave one guild and go to another, you can still see everything that you scanned.

2: Anything that you scan while in a guild will be accessible by that guild, even if you leave. For example, if you scan every astro in X10 while in Guild X, then leave for Guild Y, Guild X can still see everything that you scanned in X10.

3: Anything that you have personally scanned is accessible by everyone in your current guild. So, to use the previous example, once you leave Guild X and join Guild Y, Guild Y can see everything that you scouted in X10.

Now, while I admit that I don’t know the code to the game, I would imagine that this can be done by a few simple tricks: you would still have to maintain info per player, but by adding a server-side note that X info was scanned while in Y guild, you thus only have to add a quick check to see what guild the person viewing the DB is in. The same thing goes for the current guild; if the original scanner is in guild X, then anyone in guild X can see it.

There is still an issue of server resources, as the DB would likely have to hold info for each individual player. This would primarily be an issue on newer servers, as older ones likely don’t have more than 500 players that would actively scout and thus require heavy resources. Still, I don’t know of a way to add a DB without this problem, so as far as I’m concerned it’s one of those necessary evils that will make or break the idea.

What about spies?

I’m glad you asked! Along with the DB, I propose the following three amendments to privileges:

1: Bases (B for short) – allows players to see the bases of their guildmates. This does not affect people that have their own eyes in the region.
2: Database (D for short) – allows players to see the bases of their guildmates in the database. This does not affect people that have that info in their own collected information.
3: Fleets (F for short) - allows players to see the fleets of their guildmates in the database. This does not affect people that have that info in their own collected information.

The benefit of the first one is that a new player, or someone visiting from another guild, can’t blitz through and scout everything, and then take it back to their old guild. Likewise, someone can’t just join/visit a guild and pull their own DB entries for use against them. It’s a rather simple measure, but one that I feel is effective.

How is data added?

On every map page, there is a small bar that contains a blank list. On the system view, there is an icon next to each astro that will copy the coordinates to the list; the list cannot have coordinates added manually. If you have unlimited database functionality (see the upgrades thread for more details), you will have access to a button on the system view that automatically adds all coordinates in the system to the list.

The list stays populated, even if you click outside of the map screen; however, individual entries will be removed automatically sixty minutes after they’re added. There are two options on the list itself: once the list contains fifty or more coordinates, you can flush the list to have it update the DB with all of the info for those coordinates, or, if you have less than fifty, you can likewise flush the list and update the DB…but you cannot flush the list again for another 60 minutes. The key here is that you have to populate the list with a decent number of coordinates before updating the DB, to cut down on server hits. You can still add a smaller number, such as if you’re adding a single system, but doing so means that you cannot update the DB again for a set period of time, thus preventing too many server hits.

Note that you do not need to have eyes on the astro when you flush the list; you only need eyes when you are adding coordinates to the list. The DB updates with whatever info is there when you flush the list, so if you hang onto the coordinates for 45 minutes and something changes (ie, fleets leave or arrive) the DB will record the info from when you flushed it, not what from what you saw when you added the coordinates.

The benefit of this is also that you do not have to click into each astro; if you’re updating everything, and you pay a few dollars more for the button to add every astro, you can do an entire region in a minute or two. While some people feel this is too fast, it’s something that rivals autoscouters, and still requires people to manually go through each system in a region. I do not believe that we can cut down on cheating if we make the ability to gather data weaker than this.

Additionally, you can only update the same location once every three hours. This is to prevent people from updating the same blob every five minutes; you need multiple people doing so, which allows the same result, but cuts down on the server hits.

How long does an entry stay?

If the person adding the entry has limited functionality (again, see the upgrades thread), then the entry is deleted after seven days; if they have unlimited functionality, it is deleted after sixty days. Now, please understand how this works; as entries are timestamped, if there are multiple entries for the same location, you see the one with the most recent timestamp. Now, if your entry is deleted, but there is another, older entry (from someone with unlimited functionality) for that location, then the view switches to the older entry.

Likewise, if you go to a new guild that has an entry for a given location from yesterday, and yours is six days old, your entry will not show; when yours is deleted, nothing will change. So, just keep that in mind – it’s not a case where the data isn’t added if someone else already added it, so that you have nothing if the first guy’s entry is deleted.

How do you access the database?

Between the links for Guild and Notes would be a link to the DB. Clicking it sends you to a search page; in order to access data, you have to run a search, using the following criteria:

(Note: you cannot run a general search for every entry in a galaxy. You must select at least one other criteria. All other criteria is optional, but you can search for multiple criteria at a time)

  • Galaxy
  • Player (includes bases and fleets)
  • Guild (includes bases and fleets) (includes option for unguilded, United Colonies, and Drekons)
  • Astro type (shows only empty astros)
  • Minimum econ
  • Minimum jumpgate
  • Fleet size (single player)
  • Fleet size (total)
  • Region
  • Exclude allied bases/fleets

Note that you can also input specific, full coordinates to see if they are in the DB.

What about lag?

There was a concern voiced in the concept discussion that having data stored by player would cause too much lag. Well, remember this: one character of text = one byte of data. If every single player scouted every single astro into a database, the actual size of that data would be insignificant. The only causes of lag are inputting the data, and searching for it.

Searching, by the way, has a few easy tricks that greatly reduce the amount of lag, such as a passive stamp. When calling up all of the entries for coordinates X, you can first search by guild; by having the database store the data in the chronological order in which it was added, you run through the most recent entries for that location until you find the specified guild; the most recent entry for that guild is necessarily the one with the most recent timestamp. A quick check of the timestamp to ensure that it is not past the cutoff date for the person that added it ensures that you don’t display data from limited users past the seven day mark.

Really, I can’t offer anything more than educated guesswork; however, given the ridiculously small size of the actual data and the limitations on displaying data, I cannot believe that running databases through guilds would save a significant amount of server resources. When you consider that there are 10,796 players on Ixion, and 195,341 astros, the total data for Ixion if every single person scouted every single astro is roughly 210.9gb of data. It sounds like a lot, I know. However, consider that it would mean every single player has upgraded; if they all bought a one-month upgrade, that would be 43,076 Euros a month. If everyone bought a six-month upgrade, it’s 161,832 – just for Ixion. If they all buy a full year, it’s 215,812 Euros. That’s quite a bit of money in AE’s pocket. And remember, the Ixion scenario is an absolute worst-case scenario; there is no way that even half of the players will be scanning every single astro.

Final Thoughts:

I’m sure that there’s something I’ve missed in all of this; I’ve tried to be thorough, but my spare time of late has been practically non-existent. Anyways, I’ve done my best to handle the concerns voiced by people in the concept discussion, while still providing a database idea that is viable, useful, and can cut down on the cheating. I’m open to suggestions if people have things that they would like to see changed.

So, rather than spend the next week wondering what it is that I’ve missed – as I’m sure there’s something (I haven’t used a database in two years, so I’m a bit rusty when it comes to them) – I’ll just post this now, and see what comes of it.

Comments, questions, and concerns are welcome.
Last edited by Winchester on Wed 10 Nov, 2010 00:38, edited 1 time in total.

"That's what I do. I drink and I know things."
User avatar
kris
Platinum Member
Platinum Member
Posts: 3067
Joined: Thu 15 Mar, 2007 21:33
Reputation: 73

Re: Astro Empires Database

Postby kris » Wed 10 Nov, 2010 00:28

looks good overall however there's a few things that I think should be changed or added.

1. As far as the permissions go I don't think its detailed enough. I mean really I don't see the problem with seeing bases, in fact they are needed so people can use gates. A spy would be able to get the bases anyway, but its the fleets that shouldn't be seen since you wouldn't want one of them searching for fleets out on an op or something.

Some sort of permissions should also be added to stop allies fleets being seen too.

2.I don't see why the data needs to be deleted. If someone needs to find a gate in a particular cluster and it hasn't been scouted for a while then surely they shouldn't have to re scout the whole cluster to find what gates are suitable to launch to.

3. Search Options. Why cant you search for every entry in a galaxy? Also there should be a NOT option so you can exclude allied bases. This would help in making target lists.

User avatar
Winchester
Addicted Member
Addicted Member
Posts: 17730
Joined: Tue 24 Mar, 2009 00:44
Reputation: 766
Location: The World Wide Cesspool

Re: Astro Empires Database

Postby Winchester » Wed 10 Nov, 2010 00:35

kris wrote:looks good overall however there's a few things that I think should be changed or added.

1. As far as the permissions go I don't think its detailed enough. I mean really I don't see the problem with seeing bases, in fact they are needed so people can use gates. A spy would be able to get the bases anyway, but its the fleets that shouldn't be seen since you wouldn't want one of them searching for fleets out on an op or something.


Okay, I can see adding fleets, but I don't see a reason to remove the priv to see bases. After all, if you personally don't care, you can just give everyone the priv.

Some sort of permissions should also be added to stop allies fleets being seen too.


Hard-coding politics isn't going to happen.

2.I don't see why the data needs to be deleted. If someone needs to find a gate in a particular cluster and it hasn't been scouted for a while then surely they shouldn't have to re scout the whole cluster to find what gates are suitable to launch to.


It cuts down on server resources, and provides an incentive to get the better upgrade. If your guild can't rescout the server in sixty days, then that's really your own problem.

3. Search Options. Why cant you search for every entry in a galaxy? Also there should be a NOT option so you can exclude allied bases. This would help in making target lists.


The idea behind not searching an entire galaxy is to cut down on resources, and to not generate a huge list. Also, about excluding allied bases, good point - I'll add that.

"That's what I do. I drink and I know things."
User avatar
kris
Platinum Member
Platinum Member
Posts: 3067
Joined: Thu 15 Mar, 2007 21:33
Reputation: 73

Re: Astro Empires Database

Postby kris » Wed 10 Nov, 2010 00:48

Okay, I can see adding fleets, but I don't see a reason to remove the priv to see bases. After all, if you personally don't care, you can just give everyone the priv.


Yeah I could live with that :)

Hard-coding politics isn't going to happen.


I'm not suggesting politics be hard coded, I'm suggesting that a GM should be able to decide what can be seen by the guild database.


It cuts down on server resources, and provides an incentive to get the better upgrade. If your guild can't rescout the server in sixty days, then that's really your own problem.


No guild would want to rescout the whole server every 60 days just to keep it up to date. They'd scout when they need to, but not having lists of gates to launch will mean guilds will have to use their own databases to save the data that ae deleted. I thought the whole idea was to get rid of external databases is it not?

3. Search Options. Why cant you search for every entry in a galaxy? Also there should be a NOT option so you can exclude allied bases. This would help in making target lists.

The idea behind not searching an entire galaxy is to cut down on resources, and to not generate a huge list. Also, about excluding allied bases, good point - I'll add that.


The reason Im asking is for making target lists. to be honest it could be done with multiple searches but then wouldn't that actually use more resources?

User avatar
actimel
Platinum Member
Platinum Member
Posts: 3026
Joined: Wed 16 Aug, 2006 21:34
Reputation: 13
Guild: X.Org
Galaxy: Alpha
Location: Porto , Portugal

Re: Astro Empires Database

Postby actimel » Wed 10 Nov, 2010 00:52

this will stop major lagging problems when a big war strikes so , I am all in on this . great job

User avatar
Winchester
Addicted Member
Addicted Member
Posts: 17730
Joined: Tue 24 Mar, 2009 00:44
Reputation: 766
Location: The World Wide Cesspool

Re: Astro Empires Database

Postby Winchester » Wed 10 Nov, 2010 00:54

kris wrote:
Hard-coding politics isn't going to happen.


I'm not suggesting politics be hard coded, I'm suggesting that a GM should be able to decide what can be seen by the guild database.


Any suggestions on how you would set that up?

2.I don't see why the data needs to be deleted. If someone needs to find a gate in a particular cluster and it hasn't been scouted for a while then surely they shouldn't have to re scout the whole cluster to find what gates are suitable to launch to.


It cuts down on server resources, and provides an incentive to get the better upgrade. If your guild can't rescout the server in sixty days, then that's really your own problem.


No guild would want to rescout the whole server every 60 days just to keep it up to date. They'd scout when they need to, but not having lists of gates to launch will mean guilds will have to use their own databases to save the data that ae deleted. I thought the whole idea was to get rid of external databases is it not?


Why can't you assign a few people to search a galaxy for jumpgates, then go re-up them? However, what about extending the cutoff to ninety days? That's a rather long time for data to be up.

3. Search Options. Why cant you search for every entry in a galaxy? Also there should be a NOT option so you can exclude allied bases. This would help in making target lists.

The idea behind not searching an entire galaxy is to cut down on resources, and to not generate a huge list. Also, about excluding allied bases, good point - I'll add that.


The reason Im asking is for making target lists. to be honest it could be done with multiple searches but then wouldn't that actually use more resources?


You could search a galaxy and put in a minimum econ of 1. That would cut out all of the empty astros and 0/0 econ bases, and give you the entire galaxy.

"That's what I do. I drink and I know things."
User avatar
kris
Platinum Member
Platinum Member
Posts: 3067
Joined: Thu 15 Mar, 2007 21:33
Reputation: 73

Re: Astro Empires Database

Postby kris » Wed 10 Nov, 2010 01:11

Ribbentrop wrote:Any suggestions on how you would set that up?


You could have the permissions linked to guild ids so you can add or delete guild ids who's fleets you wouldn't want seen.


Why can't you assign a few people to search a galaxy for jumpgates, then go re-up them? However, what about extending the cutoff to ninety days? That's a rather long time for data to be up.


Well lets say I want to go to the 50s to hunt a fleet in say 56. I might want to land in 55, I might want to land in 57, there might not be any suitable gates off enemy scanners in 55 or 57 so Id need the data from the whole cluster really.

Scouting an entire cluster for a small op like this just isnt going to happen and people would get bored of doing it every time they wanted to launch somewhere just to find a gate to launch to.




You could search a galaxy and put in a minimum econ of 1. That would cut out all of the empty astros and 0/0 econ bases, and give you the entire galaxy.


Yeah that would work :)

User avatar
Callum
Volunteer
Volunteer
Posts: 4442
Joined: Wed 21 Apr, 2010 14:53
Reputation: 274
Galaxy: Alpha
Location: Great Britain

Re: Astro Empires Database

Postby Callum » Wed 10 Nov, 2010 01:13

A quick point from the third category- If you can't see data allies have added (if smaller guilds (or sub-guilds) were to share a DB) then people would still use one from outside the game, simply because the guild of 50 can't do the scouting a guild of 150 can; but they can if three guilds of 50 do.

I understand the need for not hard-coded any inter guild relations, but this is one downside that may be impossible to address without doing that.

It could go as far as interguild privs, which would not prevent them still shooting each other, but allow them to share DBs.

Edit,

kris wrote:
Ribbentrop wrote:Any suggestions on how you would set that up?


You could have the permissions linked to guild ids so you can add or delete guild ids who's fleets you wouldn't want seen.


Just like this.

That is all.
VGM of Alpha
User avatar
Universe
Gold Member
Gold Member
Posts: 1884
Joined: Mon 19 May, 2008 16:11
Reputation: 61

Re: Astro Empires Database

Postby Universe » Wed 10 Nov, 2010 02:29

I want to point out that I am still against the concept of a database.

Regardless, this is the fairest one I've seen. I would request that you not change the time limit on the entries. That is the point that sells this idea to me as an AE made database should be a scouting tool, not a scouting replacement.

"We are just an advanced breed of monkeys on a minor planet of a very average star. But we can understand the Universe. That makes us something very special." - Stephen Hawking
User avatar
kris
Platinum Member
Platinum Member
Posts: 3067
Joined: Thu 15 Mar, 2007 21:33
Reputation: 73

Re: Astro Empires Database

Postby kris » Wed 10 Nov, 2010 11:25

@actimel, I agree that we need a database but not sure how you think it will save lag during wars.

@universe how much do you play ae? What server and what guild? Does your guild not have a database?

Does that database not list JGs and enemy bases? Im pretty sure it doesnt delete therm right?

If AE makes a sub standard database that doesn't save the bases permanently what is to stop the script writers making scripts that copy down the data from the database and putting it into their own?

Seriously if the database isnt good enough for guilds that exactly what they will do. Surely anyone can see that cant they?

User avatar
Universe
Gold Member
Gold Member
Posts: 1884
Joined: Mon 19 May, 2008 16:11
Reputation: 61

Re: Astro Empires Database

Postby Universe » Wed 10 Nov, 2010 11:42

Is that your only argument? That if it doesn't have unrestricted access people will still use scripts and their own databases? I am positive people will use a database like this over the possibility of getting banned. Besides, if a large guild cannot scout a server in 2 months then it is a fail guild.

"We are just an advanced breed of monkeys on a minor planet of a very average star. But we can understand the Universe. That makes us something very special." - Stephen Hawking
User avatar
-Link-
Platinum Member
Platinum Member
Posts: 2216
Joined: Fri 05 Mar, 2010 00:56
Reputation: 29
Guild: [«ISM»]
Galaxy: Alpha
Contact:

Re: Astro Empires Database

Postby -Link- » Wed 10 Nov, 2010 11:56

guilds with databases dont suddenly stop scouting, i actually find it increases scouting as your always trying to update the info. i agree that data entered should be permanent, or atleast until it is rescouted, inwhich case delete the info and replace.. in other words, update it.

-Link- wrote:he crawls from his cave of baww where the sheep roam fluffy and freely until they plummet to their deaths at the hands of the merciless fr regulars
SpeedySurfer
Banned
Banned
Posts: 2501
Joined: Tue 13 May, 2008 21:52
Reputation: 64
Guild: M: SEXY
N: TIME
Galaxy: Nova
Location: UK

Re: Astro Empires Database

Postby SpeedySurfer » Wed 10 Nov, 2010 12:02

Universe wrote:Is that your only argument? That if it doesn't have unrestricted access people will still use scripts and their own databases? I am positive people will use a database like this over the possibility of getting banned. Besides, if a large guild cannot scout a server in 2 months then it is a fail guild.

If the guild uses a passive scanner, requiring the players to look at the bases, they won't get banned as it cannot be detected.

M: SEXY | N: TIME
User avatar
kris
Platinum Member
Platinum Member
Posts: 3067
Joined: Thu 15 Mar, 2007 21:33
Reputation: 73

Re: Astro Empires Database

Postby kris » Wed 10 Nov, 2010 12:34

Universe wrote:Is that your only argument? That if it doesn't have unrestricted access people will still use scripts and their own databases? I am positive people will use a database like this over the possibility of getting banned. Besides, if a large guild cannot scout a server in 2 months then it is a fail guild.


They wont get banned, since it wont even need players to all use a scipt. One guy could use a script to copy all the ae database to an guild database that doesnt get deleted and is accessed offline from ae.

AS for a guild scouting a server every two months being a fail guild I guess you think every guild in AE is a fail guild then because I doubt any guild scouts a whole server every two months.

You still didn't answer what server or guild you was in. Reason I ask is because I was wondering what, if any databases they use now and why you would be so anti database.

Tedex
Junior Member
Junior Member
Posts: 58
Joined: Fri 24 Jul, 2009 15:55
Reputation: 0

Re: Astro Empires Database

Postby Tedex » Wed 10 Nov, 2010 19:30

About deletion of data in DB:
- lets Info about Bases (owner, Defense, CC, JG) stays forever.(until it is rescouted)
I don't think that this info is so big to take Gigabytes of memory, and all guild DBs never deleted that data - so they do not complain about lack of resources.
Second - this way when rescouting cluster - ppl will send scouts to high JG without need first to find it - thus will reduce server load (page impressions).

The Callum wrote:A quick point from the third category- If you can't see data allies have added (if smaller guilds (or sub-guilds) were to share a DB) then people would still use one from outside the game, simply because the guild of 50 can't do the scouting a guild of 150 can; but they can if three guilds of 50 do.


I agree.
If seeing allies scouting data is impossible to code, may be only way is to increase max. members of guild. (But we can end up with 2 guilds in server).
But if there is no option to see allies scouting data - IMHO there will be aggregated user-DB which combine that info from all ailed guilds, and think that this is what we try to avoid.


Return to “FR Workshop”

Who is online

Users browsing this forum: No registered users and 1 guest