DFSOutputStream (Apache Hadoop HDFS Client 3.4.1.200-dep-1000 API)

java.lang.Object
- java.io.OutputStream
- - org.apache.hadoop.fs.FSOutputSummer
  - - org.apache.hadoop.hdfs.DFSOutputStream

All Implemented Interfaces:

java.io.Closeable, java.io.Flushable, java.lang.AutoCloseable, org.apache.hadoop.fs.CanSetDropBehind, org.apache.hadoop.fs.StreamCapabilities, org.apache.hadoop.fs.Syncable

Direct Known Subclasses:

DFSStripedOutputStream
```
@Private
public class DFSOutputStream
extends org.apache.hadoop.fs.FSOutputSummer
implements org.apache.hadoop.fs.Syncable, org.apache.hadoop.fs.CanSetDropBehind, org.apache.hadoop.fs.StreamCapabilities
```
DFSOutputStream creates files from a stream of bytes. The client application writes data that is cached internally by this stream. Data is broken up into packets, each packet is typically 64K in size. A packet comprises of chunks. Each chunk is typically 512 bytes and has an associated checksum with it. When a client application fills up the currentPacket, it is enqueued into the dataQueue of DataStreamer. DataStreamer is a thread that picks up packets from the dataQueue and sends it to the first datanode in the pipeline.

Nested Class Summary
- Nested classes/interfaces inherited from interface org.apache.hadoop.fs.StreamCapabilities
  org.apache.hadoop.fs.StreamCapabilities.StreamCapability

Field Summary

Fields
Modifier and Type	Field	Description
`protected long`	`blockSize`
`protected ByteArrayManager`	`byteArrayManager`
`protected int`	`bytesPerChecksum`
`protected java.util.concurrent.atomic.AtomicReference<CachingStrategy>`	`cachingStrategy`
`protected int`	`chunksPerPacket`
`protected boolean`	`closed`
`protected DFSPacket`	`currentPacket`
`protected DFSClient`	`dfsClient`
`protected long`	`fileId`
`protected long`	`initialFileSize`
`protected long`	`lastFlushOffset`
`protected int`	`packetSize`
`protected boolean`	`shouldSyncBlock`
`protected java.lang.String`	`src`
`protected org.apache.hadoop.hdfs.DataStreamer`	`streamer`

Fields inherited from interface org.apache.hadoop.fs.StreamCapabilities
ABORTABLE_STREAM, DROPBEHIND, HFLUSH, HSYNC, IOSTATISTICS, IOSTATISTICS_CONTEXT, PREADBYTEBUFFER, READAHEAD, READBYTEBUFFER, UNBUFFER, VECTOREDIO

Constructor Summary

Constructors
Modifier	Constructor	Description
`protected`	`DFSOutputStream(DFSClient dfsClient, java.lang.String src, HdfsFileStatus stat, java.util.EnumSet<org.apache.hadoop.fs.CreateFlag> flag, org.apache.hadoop.util.Progressable progress, org.apache.hadoop.util.DataChecksum checksum, java.lang.String[] favoredNodes, boolean createStreamer)`	Construct a new output stream for creating a file.

Method Summary

All Methods Instance Methods Concrete Methods Deprecated Methods
Modifier and Type	Method	Description
`protected void`	`adjustChunkBoundary()`	If the reopened file did not end at chunk boundary and the above write filled up its partial chunk.
`protected void`	`checkClosed()`
`void`	`close()`	Closes this output stream and releases any system resources associated with this stream.
`protected void`	`closeImpl()`
`protected void`	`closeThreads(boolean force)`
`protected void`	`completeFile(ExtendedBlock last)`
`protected void`	`computePacketChunkSize(int psize, int csize)`
`protected DFSPacket`	`createPacket(int packetSize, int chunksPerPkt, long offsetInBlock, long seqno, boolean lastPacketInBlock)`	Use `ByteArrayManager` to create buffer for non-heartbeat packets.
`protected org.apache.hadoop.tracing.TraceScope`	`createWriteTraceScope()`
`protected void`	`flushInternal()`	Waits till all existing data is flushed and confirmations received from datanodes.
`protected long`	`flushInternalWithoutWaitingAck()`
`protected java.util.EnumSet<AddBlockFlag>`	`getAddBlockFlags()`
`int`	`getCurrentBlockReplication()`	Note that this is not a public API; use `HdfsDataOutputStream.getCurrentBlockReplication()` instead.
`org.apache.hadoop.fs.FileEncryptionInfo`	`getFileEncryptionInfo()`
`long`	`getFileId()`
`long`	`getInitialLen()`	Returns the size of a file as it was when this stream was opened
`java.lang.String`	`getNamespace()`
`int`	`getNumCurrentReplicas()`	Deprecated. use `HdfsDataOutputStream.getCurrentBlockReplication()`.
`DatanodeInfo[]`	`getPipeline()`
`protected org.apache.hadoop.hdfs.DataStreamer`	`getStreamer()`	Returns the data streamer object.
`java.lang.String`	`getUniqKey()`
`boolean`	`hasCapability(java.lang.String capability)`
`void`	`hflush()`	Flushes out to all replicas of the block.
`void`	`hsync()`
`void`	`hsync(java.util.EnumSet<HdfsDataOutputStream.SyncFlag> syncFlags)`	The expected semantics is all data have flushed out to all replicas and all replicas have done posix fsync equivalent - ie the OS has flushed it to the disk device (but the disk may have it in its cache).
`protected void`	`recoverLease(boolean recoverLeaseOnCloseException)`	If recoverLeaseOnCloseException is true and an exception occurs when closing a file, recover lease.
`void`	`setArtificialSlowdown(long period)`
`void`	`setChunksPerPacket(int value)`
`void`	`setDropBehind(java.lang.Boolean dropBehind)`
`protected void`	`start()`
`void`	`sync()`	Deprecated.
`java.lang.String`	`toString()`
`protected void`	`writeChunk(byte[] b, int offset, int len, byte[] checksum, int ckoff, int cklen)`
`protected void`	`writeChunk(java.nio.ByteBuffer buffer, int len, byte[] checksum, int ckoff, int cklen)`

Methods inherited from class org.apache.hadoop.fs.FSOutputSummer
convertToByteStream, flush, flushBuffer, flushBuffer, getBufferedDataSize, getChecksumSize, getDataChecksum, resetChecksumBufSize, setChecksumBufSize, write, write

Methods inherited from class java.io.OutputStream
nullOutputStream, write

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

dfsClient
```
protected final DFSClient dfsClient
```

byteArrayManager

protected final ByteArrayManager byteArrayManager

closed
```
protected volatile boolean closed
```

src
```
protected final java.lang.String src
```

fileId
```
protected final long fileId
```

blockSize
```
protected final long blockSize
```

bytesPerChecksum
```
protected final int bytesPerChecksum
```

currentPacket
```
protected DFSPacket currentPacket
```

streamer

protected org.apache.hadoop.hdfs.DataStreamer streamer

packetSize
```
protected int packetSize
```

chunksPerPacket
```
protected int chunksPerPacket
```

lastFlushOffset
```
protected long lastFlushOffset
```

initialFileSize
```
protected long initialFileSize
```

shouldSyncBlock
```
protected boolean shouldSyncBlock
```

cachingStrategy

protected final java.util.concurrent.atomic.AtomicReference<CachingStrategy> cachingStrategy

Constructor Detail

DFSOutputStream

protected DFSOutputStream(DFSClient dfsClient,
                          java.lang.String src,
                          HdfsFileStatus stat,
                          java.util.EnumSet<org.apache.hadoop.fs.CreateFlag> flag,
                          org.apache.hadoop.util.Progressable progress,
                          org.apache.hadoop.util.DataChecksum checksum,
                          java.lang.String[] favoredNodes,
                          boolean createStreamer)

Construct a new output stream for creating a file.

Method Detail
- createPacket
```
protected DFSPacket createPacket(int packetSize,
                                 int chunksPerPkt,
                                 long offsetInBlock,
                                 long seqno,
                                 boolean lastPacketInBlock)
                          throws java.io.InterruptedIOException
```
  Use ByteArrayManager to create buffer for non-heartbeat packets.
  
  Throws:
  
  java.io.InterruptedIOException
- checkClosed
```
protected void checkClosed()
                    throws java.io.IOException
```
  Specified by:
  
  checkClosed in class org.apache.hadoop.fs.FSOutputSummer
  
  Throws:
  
  java.io.IOException
- getPipeline
```
@VisibleForTesting
public DatanodeInfo[] getPipeline()
```
- computePacketChunkSize
```
protected void computePacketChunkSize(int psize,
                                      int csize)
```
- createWriteTraceScope
```
protected org.apache.hadoop.tracing.TraceScope createWriteTraceScope()
```
  Overrides:
  
  createWriteTraceScope in class org.apache.hadoop.fs.FSOutputSummer
- writeChunk
```
protected void writeChunk(byte[] b,
                          int offset,
                          int len,
                          byte[] checksum,
                          int ckoff,
                          int cklen)
                   throws java.io.IOException
```
  Specified by:
  
  writeChunk in class org.apache.hadoop.fs.FSOutputSummer
  
  Throws:
  
  java.io.IOException
- writeChunk
```
protected void writeChunk(java.nio.ByteBuffer buffer,
                          int len,
                          byte[] checksum,
                          int ckoff,
                          int cklen)
                   throws java.io.IOException
```
  Throws:
  
  java.io.IOException
- adjustChunkBoundary
```
protected void adjustChunkBoundary()
```
  If the reopened file did not end at chunk boundary and the above write filled up its partial chunk. Tell the summer to generate full crc chunks from now on.
- hasCapability
```
public boolean hasCapability(java.lang.String capability)
```
  Specified by:
  
  hasCapability in interface org.apache.hadoop.fs.StreamCapabilities
  
  Overrides:
  
  hasCapability in class org.apache.hadoop.fs.FSOutputSummer
- hflush
```
public void hflush()
            throws java.io.IOException
```
  Flushes out to all replicas of the block. The data is in the buffers of the DNs but not necessarily in the DN's OS buffers. It is a synchronous operation. When it returns, it guarantees that flushed data become visible to new readers. It is not guaranteed that data has been flushed to persistent store on the datanode. Block allocations are persisted on namenode.
  
  Specified by:
  
  hflush in interface org.apache.hadoop.fs.Syncable
  
  Throws:
  
  java.io.IOException
- hsync
```
public void hsync()
           throws java.io.IOException
```
  Specified by:
  
  hsync in interface org.apache.hadoop.fs.Syncable
  
  Throws:
  
  java.io.IOException
- sync
```
@Deprecated
public void sync()
          throws java.io.IOException
```
  Deprecated.
  
  Specified by:
  
  sync in interface org.apache.hadoop.fs.Syncable
  
  Throws:
  
  java.io.IOException
- hsync
```
public void hsync(java.util.EnumSet<HdfsDataOutputStream.SyncFlag> syncFlags)
           throws java.io.IOException
```
  The expected semantics is all data have flushed out to all replicas and all replicas have done posix fsync equivalent - ie the OS has flushed it to the disk device (but the disk may have it in its cache). Note that only the current block is flushed to the disk device. To guarantee durable sync across block boundaries the stream should be created with CreateFlag.SYNC_BLOCK.
  
  Parameters:
  
  syncFlags - Indicate the semantic of the sync. Currently used to specify whether or not to update the block length in NameNode.
  
  Throws:
  
  java.io.IOException
- getNumCurrentReplicas
```
@Deprecated
public int getNumCurrentReplicas()
                          throws java.io.IOException
```
  Deprecated.
  use HdfsDataOutputStream.getCurrentBlockReplication().
  
  Throws:
  
  java.io.IOException
- getCurrentBlockReplication
```
public int getCurrentBlockReplication()
                               throws java.io.IOException
```
  Note that this is not a public API; use HdfsDataOutputStream.getCurrentBlockReplication() instead.
  
  Returns:
  
  the number of valid replicas of the current block
  
  Throws:
  
  java.io.IOException
- flushInternal
```
protected void flushInternal()
                      throws java.io.IOException
```
  Waits till all existing data is flushed and confirmations received from datanodes.
  
  Throws:
  
  java.io.IOException
- start
```
protected void start()
```
- closeThreads
```
protected void closeThreads(boolean force)
                     throws java.io.IOException
```
  Throws:
  
  java.io.IOException
- close
```
public void close()
           throws java.io.IOException
```
  Closes this output stream and releases any system resources associated with this stream.
  
  Specified by:
  
  close in interface java.lang.AutoCloseable
  
  Specified by:
  
  close in interface java.io.Closeable
  
  Overrides:
  
  close in class java.io.OutputStream
  
  Throws:
  
  java.io.IOException
- closeImpl
```
protected void closeImpl()
                  throws java.io.IOException
```
  Throws:
  
  java.io.IOException
- recoverLease
```
protected void recoverLease(boolean recoverLeaseOnCloseException)
```
  If recoverLeaseOnCloseException is true and an exception occurs when closing a file, recover lease.
- completeFile
```
protected void completeFile(ExtendedBlock last)
                     throws java.io.IOException
```
  Throws:
  
  java.io.IOException
- setArtificialSlowdown
```
@VisibleForTesting
public void setArtificialSlowdown(long period)
```
- setChunksPerPacket
```
@VisibleForTesting
public void setChunksPerPacket(int value)
```
- getInitialLen
```
public long getInitialLen()
```
  Returns the size of a file as it was when this stream was opened
- getAddBlockFlags
```
protected java.util.EnumSet<AddBlockFlag> getAddBlockFlags()
```
- getFileEncryptionInfo
```
public org.apache.hadoop.fs.FileEncryptionInfo getFileEncryptionInfo()
```
  Returns:
  
  the FileEncryptionInfo for this stream, or null if not encrypted.
- flushInternalWithoutWaitingAck
```
protected long flushInternalWithoutWaitingAck()
                                       throws java.io.IOException
```
  Throws:
  
  java.io.IOException
- setDropBehind
```
public void setDropBehind(java.lang.Boolean dropBehind)
                   throws java.io.IOException
```
  Specified by:
  
  setDropBehind in interface org.apache.hadoop.fs.CanSetDropBehind
  
  Throws:
  
  java.io.IOException
- getFileId
```
@VisibleForTesting
public long getFileId()
```
- getNamespace
```
@VisibleForTesting
public java.lang.String getNamespace()
```
- getUniqKey
```
@VisibleForTesting
public java.lang.String getUniqKey()
```
- getStreamer
```
protected org.apache.hadoop.hdfs.DataStreamer getStreamer()
```
  Returns the data streamer object.
- toString
```
public java.lang.String toString()
```
  Overrides:
  
  toString in class java.lang.Object

Class DFSOutputStream

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.hadoop.fs.StreamCapabilities

Field Summary

Fields inherited from interface org.apache.hadoop.fs.StreamCapabilities

Constructor Summary

Method Summary

Methods inherited from class org.apache.hadoop.fs.FSOutputSummer

Methods inherited from class java.io.OutputStream

Methods inherited from class java.lang.Object

Field Detail

dfsClient

byteArrayManager

closed

src

fileId

blockSize

bytesPerChecksum

currentPacket

streamer

packetSize

chunksPerPacket

lastFlushOffset

initialFileSize

shouldSyncBlock

cachingStrategy

Constructor Detail

DFSOutputStream

Method Detail

createPacket

checkClosed

getPipeline

computePacketChunkSize

createWriteTraceScope

writeChunk

writeChunk

adjustChunkBoundary

hasCapability

hflush

hsync

sync

hsync

getNumCurrentReplicas

getCurrentBlockReplication

flushInternal

start

closeThreads

close

closeImpl

recoverLease

completeFile

setArtificialSlowdown

setChunksPerPacket

getInitialLen

getAddBlockFlags

getFileEncryptionInfo

flushInternalWithoutWaitingAck

setDropBehind

getFileId

getNamespace

getUniqKey

getStreamer

toString