I wondered why VeeshanVault ran so dang slow...

Diae Soulmender · Post by **Diae Soulmender** » July 10, 2003, 6:57 pm

I work for a large west coast ISP. I ran some tests to veeshanvault and found some interesting items that I thought I would bring to Xou attention.

Keep in mind Im behind multiple redundent DS3's running these tests behind our Portland, OR Edge router which is a Cisco 7500 Series...

First I ran 1000 packets at 1500 bytes each to your server from my Big Dog Daddy Phat Router:

I got tired of waiting on them 1000 packets so I stopped the pings... notice anything wrong? :)

Protocol [ip]:
Target IP address: 66.193.174.75
Repeat count [5]: 1000
Datagram size [100]: 1500
Timeout in seconds [2]:
Extended commands [n]:
Sweep range of sizes [n]:
Type escape sequence to abort.
Sending 1000, 1500-byte ICMP Echos to 66.193.174.75, timeout is 2 seconds:
!.!!.!.!!.!!!!!!!.!!!!!..!!!!!!!!!!.!!!!!!!!!!!.!!!!!!!!!!!!.!!!!!!!!!
!!!!!!!!!.!!!.!!!!!!!!!!.!!!!!!!.!.!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!.!!!
!!!!!!!!.!!!..!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!..!!!!!!!!!!!!!!!.!!!!.!!!
!!!!..!.!!!!!..!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!
!!!!!!!!!!.!!!!!!.!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Success rate is 91 percent (380/415), round-trip min/avg/max = 144/431/1132 ms

9% packet loss... ouchie.

====================================

I then ran some traceroutes from our edge router to 66.193.174.75

1 *.*.*.* 0 msec 4 msec 0 msec (removed for privacy)
2 *.*.*.* [AS****] 4 msec 0 msec 0 msec (removed for privacy)
3 129.250.55.121 [AS 2914] 0 msec 0 msec 4 msec
4 129.250.55.95 [AS 2914] 0 msec 0 msec 8 msec
5 129.250.31.222 [AS 2914] 4 msec 0 msec 4 msec
6 129.250.4.31 [AS 2914] 16 msec 16 msec 16 msec
7 198.32.200.49 [AS 2914] 16 msec 16 msec 20 msec
8 66.192.250.33 [AS 4323] 20 msec 16 msec 20 msec
9 168.215.55.81 [AS 4323] 36 msec 32 msec 32 msec
10 66.192.255.33 [AS 4323] 88 msec 84 msec 88 msec
11 66.192.243.6 [AS 4323] 100 msec 100 msec 100 msec
12 66.192.243.138 [AS 4323] 100 msec 104 msec 104 msec
13 66.192.143.158 [AS 4323] 108 msec 116 msec 116 msec
14 * * *
15 * * *
16 * * *
17 * * *
18 * * *
19 * * *

Now hops 14-19 etc... dont nessasarily bother me or you except from my workstation its a little different because its taking a different weighted route our our edge...

==============================

Here are my pings from my workstation to 66.193.174.75

Ping statistics for 66.193.174.75:
Packets: Sent = 201, Received = 198, Lost = 3 (1% loss),
Approximate round trip times in milli-seconds:
Minimum = 100ms, Maximum = 601ms, Average = 134ms

Hmmmm 1% loss, that not too bad, but still can cause some nasty latency.

Heres traceroute from my workstation to 66.193.174.75

Tracing route to 66.193.174.75 over a maximum of 30 hops

1 10 ms <10 ms <10 ms *.*.*.* (removed for privacy)
2 <10 ms <10 ms 10 ms *.*.*.* (removed for privacy)
3 271 ms 10 ms 140 ms *.*.*.* (removed for privacy)
4 <10 ms 10 ms 10 ms 157.238.24.117
5 <10 ms <10 ms 10 ms 129.250.55.121
6 <10 ms 10 ms <10 ms 129.250.55.95
7 10 ms 10 ms 20 ms 129.250.31.222
8 20 ms 30 ms 20 ms 129.250.4.31
9 20 ms 20 ms 30 ms 198.32.200.49
10 20 ms 30 ms 60 ms 66.192.250.33
11 30 ms 30 ms 30 ms 168.215.55.81
12 80 ms 90 ms 90 ms 66.192.255.33
13 100 ms 110 ms 100 ms 66.192.243.6
14 100 ms 111 ms 100 ms 66.192.243.138
15 120 ms 110 ms 121 ms 66.192.143.162
16 130 ms 150 ms 120 ms 66.193.174.75

Notice the increase in ping on 12-16? Thats yer isp to your server....

Now them pings aint bad at all. Heck anything sub 150ms is just dandy really, but that packet loss, even at 1% is no good. Something is wrong Xou.

Whats even worse is the trace directly from my edge router which goes upwards or 9-10% packet loss which is TRUELY nasty.

Just thought I would point it out to you.

Feel free totrash can this but I thought ya might wanna know :)

Post by **pyrella** » July 10, 2003, 7:06 pm

K, I'll let Xou handle it.

Post by **pyrella** » July 10, 2003, 7:15 pm

Actually, here's a bit more in-depth on the situation, even though it's been mentioned a few times before.

We are currently on a dual T-1 setup(not enough bandwidth by far). When we went with this colo they had a DS-3 and things were looking great. A day or 2 after we got the machine, there bandwidth provider went bankrupt, and we had to fall back on there backup system (the dual T-1's). After way too much drama and time where Time Warner decided that there current facilities wouldn't support the OC-48, they (our co-lo) have purchased or leased facilities accross the street, have the Sonet installed and the line there and have just completed building and powering the new racks in the new location. We are currently on the TW subnet (66.x.x.x) vs. the old one (216.x.x.x), and all we are waiting for is the switch to be flipped essentially. As of yesterday for the first time I was given a confirmed date of when we will be switched to the OC-48, that date is the 15th and should hopefully get us up to acceptable speeds and then some.

As for the packet loss, I'd attribute it to packets timing out since the T-1's can't push out enough for all of the co-lo's customers at once. I'm not a complete expert on routing and such like that, and if you have something that looks like the co-lo has a router boo-boo and can give it to me in a way that I can present to them, they are usually very fast in addressing any issues that have arisen.

Diae Soulmender · Post by **Diae Soulmender** » July 10, 2003, 7:42 pm

AHHHHH that explains it very well. Yes, its not 'true' packet loss Im seeing then as its more of a "CIR" issue where you have met the bandwidth and the telco is dropping the packets on there end. This is understood.

Cant wait for you guys to go to your new facilities!

:)

masteen · Post by **masteen** » July 10, 2003, 8:56 pm

3 mbps isn't a lot of bandwidth. I used to hog that much leeching pr0n or sharing mp3's thru a cable modem.

Winnow · Post by **Winnow** » July 10, 2003, 9:22 pm

masteen wrote:3 mbps isn't a lot of bandwidth. I used to hog that much leeching pr0n or sharing mp3's thru a cable modem.

Never doubt Masteens pr0n skillz

Post by **Xouqoa** » July 10, 2003, 10:09 pm

Thanks for the info, but that's like reading latin to me. =)

But my computer works now.. yay!

Marbus · Post by **Marbus** » July 11, 2003, 1:01 am

Well even 256K PVC would probably handle this board if that was the only thing on there. But with all the other sites we are probably getting less than a DS0 most of the time. Xou, it will get better on the 15th

Marb

Post by **pyrella** » July 11, 2003, 8:29 pm

Got hit with a worm today too! .cinik to be precise - good thing we have the VV alert squad so it was fixed within a couple hours =P

We cleaned it out and SSL is being patched/updated to remove the vulnaribility - hurraj

Sargeras · Post by **Sargeras** » July 11, 2003, 8:42 pm

I want to be like Py when I grow up

noel · Post by **noel** » July 11, 2003, 8:45 pm

pyrella wrote:Got hit with a worm today too! .cinik to be precise - good thing we have the VV alert squad so it was fixed within a couple hours =P

We cleaned it out and SSL is being patched/updated to remove the vulnaribility - hurraj

Good job team!

Ebumar · Post by **Ebumar** » July 11, 2003, 10:35 pm

/golf clap

Neost · Post by **Neost** » July 12, 2003, 8:14 pm

I think we should break Diae's "NO I WILL NOT FIX YOUR FUCKING COMPUTER" mug.....

Voronwë · Post by **Voronwë** » July 14, 2003, 11:12 am

hey guys, whats going on on this thread?

Post by **pyrella** » July 14, 2003, 5:14 pm

Update:

The line installation was finished today (1 day early!) but they have to finish building the racks and running power to where they couldn't work due to Time Warner running the line.

That leaves 1-2 days for that, and another 1-2 days for them to install there own hardware (routers/switches/etc), so we are looking at Thursday - Saturday for our box to be put in the new location and on to more crap we can put up! =P

stay tuned~