sarum90

object_store EOF retry bug causing data duplication

Bug Summary

If a reqwest timeout occurs after consuming all bytes from an S3 response body but before receiving the EOF signal, retry_stream attempts to retry with a Range header like bytes=5-4 (where 5 is the file size). This is an invalid/backwards range.

Per RFC 7233, servers MUST ignore syntactically invalid Range headers. S3 follows this spec and returns 200 OK with the full file instead of 206 Partial Content or 416 Range Not Satisfiable.

Since retry_stream doesn't validate that it received a 206 response, it reads the full file again, causing data duplication.

sarum90

Test thing

Here is my preamble paragraph, but I could really place the TOC anywhere! Lorem ipsum foo bar baz.

sarum90

import timeit

_ATOMIC_TYPES_LIST = [bool, int, float, str, unicode]
_ATOMIC_TYPES_SET = set(_ATOMIC_TYPES_LIST)


def is_a_type(x):
    t = type(x)
    if (t is bool or

sarum90

from flocker.control._persistence import wire_decode, wire_encode
from collections import deque
import json
from pyrsistent import PRecord, PVector, PMap, PSet, PClass
from twisted.python.filepath import FilePath
from uuid import UUID
from datetime import datetime
from calendar import timegm
import timeit
from repoze.lru import LRUCache

sarum90

Before:

● ~/flocker (master)$ sudo mount | grep bind
/home/ubuntu/flocker/_trial_temp/flocker.node.agents.test.test_bl/BlockDeviceManagerTests/test_bind_mount/tmpk5d9T1/temp/0386b1ff-34ef-41ef-bb24-d0bfa04c048e on /home/ubuntu/flocker/_trial_temp/flocker.node.agents.test.test_bl/BlockDeviceManagerTests/test_bind_mount/tmpk5d9T1/temp/ecf14368-2697-44d1-baba-a0d785ecc45f type none (rw,bind)
● ~/flocker (master)$ sudo `which trial` -e flocker.node.agents.test.test_blockdevice_manager
could not remove FilePath('/home/ubuntu/flocker/_trial_temp'), caught OSError [Errno 16]: Device or resource busy
flocker.node.agents.test.test_blockdevice_manager
  BlockDeviceManagerTests
    test_bind_mount ...                                                    [OK]
    test_failing_bind_mount ...                                            [OK]

sarum90

from twisted.internet import reactor
from twisted.internet.task import Clock, deferLater
from twisted.internet.defer import succeed


def opaque_animal_noise_generator():
    """
    Opaque piece of test infra that we can't touch (actually does networking or
    something so really uses the real reactor).
    """

sarum90

[global]
direct=1
ioengine=libaio
rw=randwrite
numjobs=1
iodepth=256
size=50G
bs=4k
time_based
runtime=20

sarum90

[global]
direct=1
ioengine=libaio
rw=randwrite
numjobs=16
size=1G
bs=16k
time_based
runtime=20
group_reporting

sarum90

[global]
direct=1
ioengine=libaio
rw=randread
size=1G
numjobs=2
bs=4k
time_based
runtime=10
ramp_time=5

sarum90

[global]
direct=1
ioengine=libaio
rw=randwrite
numjobs=16
size=1G
bs=16k
time_based
runtime=20
group_reporting