KisPlaybackEngineMLT Class Reference

The KisPlaybackEngineMLT class is an implementation of KisPlaybackEngine that uses MLT (Media Lovin' Toolkit) to drive image frame changes and animation audio with (hopefully) close to frame-perfect synchronization. More...

#include <KisPlaybackEngineMLT.h>

Inheritance diagram for KisPlaybackEngineMLT:

Classes
struct	FrameWaitingInterface
struct	Private
struct	StopAndResume
	The StopAndResumeConsumer struct is used to encapsulate optional stop-and-then-resume behavior of a consumer. Using RAII, we can stop a consumer at construction and simply resume it when it exits scope. More...

Public Slots
bool	isMute () override
PlaybackStats	playbackStatistics () const override
void	seek (int frameIndex, SeekOptionFlags flags=SEEK_FINALIZE|SEEK_PUSH_AUDIO) override
void	setDropFramesMode (bool value) override
void	setMute (bool val) override
bool	supportsAudio () override
bool	supportsVariablePlaybackSpeed () override
Public Slots inherited from KisPlaybackEngine
bool	dropFrames () const
virtual void	firstFrame ()
virtual bool	isMute ()=0
virtual void	lastFrame ()
virtual void	nextFrame ()
virtual void	nextKeyframe ()
virtual void	nextMatchingKeyframe ()
virtual void	nextUnfilteredKeyframe ()
virtual void	pause ()
virtual void	play ()
virtual PlaybackStats	playbackStatistics () const =0
virtual void	playPause ()
virtual void	previousFrame ()
virtual void	previousKeyframe ()
virtual void	previousMatchingKeyframe ()
	previousMatchingKeyframe && nextMatchingKeyframe Navigate to the next keyframe that has the same color-label as the current keyframe. Useful to quickly navigate to user-specified 'similar' keyframes. E.g. Contact points in an animation might have a specific color to specify importance and be quickly swapped between.
virtual void	previousUnfilteredKeyframe ()
	previousUnfilteredKeyframe && nextUnfilteredKeyframe Navigate to keyframes based on the current onion skin filtration. This lets users easily navigate to the next visible "onion-skinned" keyframe on the active layer.
virtual void	seek (int frameIndex, SeekOptionFlags options=SEEK_FINALIZE|SEEK_PUSH_AUDIO)=0
virtual void	setDropFramesMode (bool value)
virtual void	setMute (bool val)=0
virtual void	stop ()
virtual bool	supportsAudio ()=0
virtual bool	supportsVariablePlaybackSpeed ()=0

Signals
void	sigChangeActiveCanvasFrame (int p_frame)
Signals inherited from KisPlaybackEngine
void	sigDropFramesModeChanged (bool value)

Public Member Functions
FrameWaitingInterface *	frameWaitingInterface ()
	KisPlaybackEngineMLT (QObject *parent=nullptr)
	~KisPlaybackEngineMLT ()
Public Member Functions inherited from KisPlaybackEngine
	KisPlaybackEngine (QObject *parent=nullptr)
	~KisPlaybackEngine ()
Public Member Functions inherited from KoCanvasObserverBase
	KoCanvasObserverBase ()
KoCanvasBase *	observedCanvas () const
virtual QString	observerName ()
void	setObservedCanvas (KoCanvasBase *canvas)
void	unsetObservedCanvas ()
virtual	~KoCanvasObserverBase ()

Protected Slots
void	canvasDestroyed (QObject *canvas)
void	setAudioVolume (qreal volumeNormalized)
	setAudioVolume
void	setCanvas (KoCanvasBase *canvas) override
void	throttledSetSpeed (const double speed)
	throttledSetSpeed
void	throttledShowFrame (const int frame)
	throttledShowFrame
void	unsetCanvas () override
Protected Slots inherited from KisPlaybackEngine
virtual void	setCanvas (KoCanvasBase *p_canvas) override
virtual void	unsetCanvas () override

Private Member Functions
void	setupProducer (boost::optional< QFileInfo > file)
	Sets up an MLT::Producer object in response to audio being added to a Krita document or when canvas changes.

Private Attributes
QScopedPointer< Private >	m_d

Additional Inherited Members
Protected Member Functions inherited from KisPlaybackEngine
class KisCanvas2 *	activeCanvas () const
int	frameWrap (int frame, int startFrame, int endFrame)
Protected Member Functions inherited from KoCanvasObserverBase
virtual void	setCanvas (KoCanvasBase *canvas)=0
virtual void	unsetCanvas ()=0

Detailed Description

The KisPlaybackEngineMLT class is an implementation of KisPlaybackEngine that uses MLT (Media Lovin' Toolkit) to drive image frame changes and animation audio with (hopefully) close to frame-perfect synchronization.

If MLT is unavailable or unwanted, Krita can instead use KisPlaybackEngineQT which may be simpler but has different characteristics and is not designed with audio-video synchronization in mind.

Definition at line 35 of file KisPlaybackEngineMLT.h.

Constructor & Destructor Documentation

KisPlaybackEngineMLT()

KisPlaybackEngineMLT::KisPlaybackEngineMLT ( QObject * parent = nullptr )

explicit

Definition at line 367 of file KisPlaybackEngineMLT.cpp.

    : KisPlaybackEngine(parent)
    , m_d( new Private(this))
{
    connect(this, &KisPlaybackEngineMLT::sigChangeActiveCanvasFrame, this, &KisPlaybackEngineMLT::throttledShowFrame, Qt::UniqueConnection);
}

References connect(), sigChangeActiveCanvasFrame(), and throttledShowFrame().

~KisPlaybackEngineMLT()

KisPlaybackEngineMLT::~KisPlaybackEngineMLT

(

)

Definition at line 374 of file KisPlaybackEngineMLT.cpp.

{
}

Member Function Documentation

canvasDestroyed

void KisPlaybackEngineMLT::canvasDestroyed ( QObject * canvas )

protectedslot

We cannot use QMap::remove here, because the canvas is already half-destroyed and we cannot up-cast to KisCanvas2 anymore

Definition at line 533 of file KisPlaybackEngineMLT.cpp.

{
    KIS_SAFE_ASSERT_RECOVER_RETURN(m_d->activeCanvas() != canvas);
 
    for (auto it = m_d->canvasProducers.begin(); it != m_d->canvasProducers.end(); ++it) {
        if (it.key() == canvas) {
            m_d->canvasProducers.erase(it);
            break;
        }
    }
}

References KIS_SAFE_ASSERT_RECOVER_RETURN, and m_d.

frameWaitingInterface()

KisPlaybackEngineMLT::FrameWaitingInterface * KisPlaybackEngineMLT::frameWaitingInterface

(

)

Definition at line 590 of file KisPlaybackEngineMLT.cpp.

{
    return &m_d->frameWaitingInterface;
}

References m_d.

isMute

bool KisPlaybackEngineMLT::isMute ( )

overrideslot

Definition at line 613 of file KisPlaybackEngineMLT.cpp.

{
    return m_d->mute;
}

References m_d.

playbackStatistics

KisPlaybackEngine::PlaybackStats KisPlaybackEngineMLT::playbackStatistics ( ) const

overrideslot

Definition at line 618 of file KisPlaybackEngineMLT.cpp.

{
    KisPlaybackEngine::PlaybackStats stats;
 
    if (activeCanvas() && activeCanvas()->animationState() &&
        m_d->activePlaybackMode() == PLAYBACK_PULL ) {
 
        const int droppedFrames = m_d->frameStats.droppedFramesCount.rollingSum();
        const int totalFrames =
            m_d->frameStats.droppedFramesCount.rollingCount() +
            droppedFrames;
 
        stats.droppedFramesPortion = qreal(droppedFrames) / totalFrames;
        stats.expectedFps = qreal(activeCanvas()->image()->animationInterface()->framerate()) * m_d->playbackSpeed;
 
        const qreal avgTimePerFrame = m_d->frameStats.averageFrameDuration.rollingMeanSafe();
        stats.realFps = !qFuzzyIsNull(avgTimePerFrame) ? 1000.0 / avgTimePerFrame : 0.0;
 
    }
 
    return stats;
}

References KisPlaybackEngine::activeCanvas(), KisPlaybackEngine::PlaybackStats::droppedFramesPortion, KisPlaybackEngine::PlaybackStats::expectedFps, m_d, PLAYBACK_PULL, qFuzzyIsNull(), and KisPlaybackEngine::PlaybackStats::realFps.

seek

void KisPlaybackEngineMLT::seek ( int frameIndex, SeekOptionFlags flags = SEEK_FINALIZE | SEEK_PUSH_AUDIO )

overrideslot

Definition at line 378 of file KisPlaybackEngineMLT.cpp.

{
    KIS_ASSERT(activeCanvas() && activeCanvas()->animationState());
    KisCanvasAnimationState* animationState = activeCanvas()->animationState();
 
    if (m_d->activePlaybackMode() == PLAYBACK_PUSH) {
        m_d->canvasProducers[activeCanvas()]->seek(frameIndex);
 
        if (flags & SEEK_PUSH_AUDIO) {
 
            m_d->sigPushAudioCompressor->start(frameIndex);
        }
 
        animationState->showFrame(frameIndex, (flags & SEEK_FINALIZE) > 0);
    }
}

References KisPlaybackEngine::activeCanvas(), KisCanvas2::animationState(), KIS_ASSERT, m_d, PLAYBACK_PUSH, SEEK_FINALIZE, SEEK_PUSH_AUDIO, and KisCanvasAnimationState::showFrame().

setAudioVolume

void KisPlaybackEngineMLT::setAudioVolume ( qreal volumeNormalized )

protectedslot

setAudioVolume

Parameters

volume

(normalized)

Definition at line 579 of file KisPlaybackEngineMLT.cpp.

{
    if (m_d->mute) {
        m_d->pullConsumer->set("volume", 0.0);
        m_d->pushConsumer->set("volume", 0.0);
    } else {
        m_d->pullConsumer->set("volume", volumeNormalized);
        m_d->pushConsumer->set("volume", volumeNormalized);
    }
}

References m_d.

setCanvas

void KisPlaybackEngineMLT::setCanvas ( KoCanvasBase * canvas )

overrideprotectedslot

Both, producer and consumers store frame rate in their private properties and do not track updates in the profile, so we should recreate all the three to actually make the changes to take effect.

Theoretically, we could just call set_profile on all the three, but the fact that we embed one more producer into our own one, prevents it from working.

Definition at line 438 of file KisPlaybackEngineMLT.cpp.

{
    KisCanvas2* canvas = dynamic_cast<KisCanvas2*>(p_canvas);
 
    if (activeCanvas() == canvas) {
        return;
    }
 
    if (activeCanvas()) {
        KisCanvasAnimationState* animationState = activeCanvas()->animationState();
 
        // Disconnect old canvas, prepare for new one..
        if (animationState) {
            this->disconnect(animationState);
            animationState->disconnect(this);
        }
 
        // Disconnect old image, prepare for new one..
        auto image = activeCanvas()->image();
        if (image && image->animationInterface()) {
            this->disconnect(image->animationInterface());
            image->animationInterface()->disconnect(this);
        }
    }
 
    StopAndResume stopResume(m_d.data(), true);
 
    KisPlaybackEngine::setCanvas(p_canvas);
 
    // Connect new canvas..
    if (activeCanvas()) {
        KisCanvasAnimationState* animationState = activeCanvas()->animationState();
        KIS_SAFE_ASSERT_RECOVER_RETURN(animationState);
 
        connect(animationState, &KisCanvasAnimationState::sigPlaybackStateChanged, this, [this](PlaybackState state){
            Q_UNUSED(state); // We don't need the state yet -- we just want to stop and resume playback according to new state info.
            StopAndResume callbackStopResume(m_d.data());
        });
 
        connect(animationState, &KisCanvasAnimationState::sigPlaybackMediaChanged, this, [this](){
            KisCanvasAnimationState* animationState = activeCanvas()->animationState();
            if (animationState) {
                setupProducer(animationState->mediaInfo());
            }
        });
 
        connect(animationState, &KisCanvasAnimationState::sigPlaybackSpeedChanged, this, [this](qreal value){
            m_d->sigSetPlaybackSpeed->start(value);
        });
        m_d->playbackSpeed = animationState->playbackSpeed();
 
        connect(animationState, &KisCanvasAnimationState::sigAudioLevelChanged, this, &KisPlaybackEngineMLT::setAudioVolume);
 
        auto image = activeCanvas()->image();
        KIS_SAFE_ASSERT_RECOVER_RETURN(image);
 
        // Connect new image..
        connect(image->animationInterface(), &KisImageAnimationInterface::sigFramerateChanged, this, [this](){
            StopAndResume callbackStopResume(m_d.data());
            m_d->profile->set_frame_rate(activeCanvas()->image()->animationInterface()->framerate(), 1);
 
            KisCanvasAnimationState* animationState = activeCanvas()->animationState();
            if (animationState) {
                setupProducer(animationState->mediaInfo());
            }
        });
 
        // cold init the framerate
        m_d->profile->set_frame_rate(activeCanvas()->image()->animationInterface()->framerate(), 1);
 
        connect(image->animationInterface(), &KisImageAnimationInterface::sigPlaybackRangeChanged, this, [this](){
            QSharedPointer<Mlt::Producer> producer = m_d->canvasProducers[activeCanvas()];
            auto image = activeCanvas()->image();
            KIS_SAFE_ASSERT_RECOVER_RETURN(image);
            producer->set("start_frame", image->animationInterface()->activePlaybackRange().start());
            producer->set("end_frame", image->animationInterface()->activePlaybackRange().end());
        });
 
        setupProducer(animationState->mediaInfo());
    }
 
}

References KisPlaybackEngine::activeCanvas(), KisCanvas2::animationState(), connect(), KisCanvas2::image(), KIS_SAFE_ASSERT_RECOVER_RETURN, m_d, KisCanvasAnimationState::mediaInfo(), KisCanvasAnimationState::playbackSpeed(), setAudioVolume(), KisPlaybackEngine::setCanvas(), setupProducer(), KisCanvasAnimationState::sigAudioLevelChanged(), KisImageAnimationInterface::sigFramerateChanged(), KisCanvasAnimationState::sigPlaybackMediaChanged(), KisImageAnimationInterface::sigPlaybackRangeChanged(), KisCanvasAnimationState::sigPlaybackSpeedChanged(), KisCanvasAnimationState::sigPlaybackStateChanged(), and value().

setDropFramesMode

void KisPlaybackEngineMLT::setDropFramesMode ( bool value )

overrideslot

Definition at line 595 of file KisPlaybackEngineMLT.cpp.

{
    // restart playback if it was active
    StopAndResume r(m_d.data(), false);
 
    KisPlaybackEngine::setDropFramesMode(value);
}

References m_d, KisPlaybackEngine::setDropFramesMode(), and value().

setMute

void KisPlaybackEngineMLT::setMute ( bool val )

overrideslot

Definition at line 603 of file KisPlaybackEngineMLT.cpp.

{
    KIS_SAFE_ASSERT_RECOVER_RETURN(activeCanvas() && activeCanvas()->animationState());
    KisCanvasAnimationState* animationState = activeCanvas()->animationState();
 
    qreal currentVolume = animationState->currentVolume();
    m_d->mute = val;
    setAudioVolume(currentVolume);
}

References KisPlaybackEngine::activeCanvas(), KisCanvas2::animationState(), KisCanvasAnimationState::currentVolume(), KIS_SAFE_ASSERT_RECOVER_RETURN, m_d, and setAudioVolume().

setupProducer()

void KisPlaybackEngineMLT::setupProducer ( boost::optional< QFileInfo > file )

private

Sets up an MLT::Producer object in response to audio being added to a Krita document or when canvas changes.

Parameters

file	An optional file to be loaded by MLT.

Definition at line 395 of file KisPlaybackEngineMLT.cpp.

{
    if (!m_d->canvasProducers.contains(activeCanvas())) {
        connect(activeCanvas(), SIGNAL(destroyed(QObject*)), this, SLOT(canvasDestroyed(QObject*)));
    }
 
    //First, assign to "count" producer.
    m_d->canvasProducers[activeCanvas()] = QSharedPointer<Mlt::Producer>(new Mlt::Producer(*m_d->profile, "krita_play_chunk", "count"));
 
    //If we have a file and the file has a valid producer, use that. Otherwise, stick to our "default" producer.
    if (file.has_value()) {
        QSharedPointer<Mlt::Producer> producer(
 
#ifdef Q_OS_ANDROID
            new Mlt::Producer(*m_d->profile,
                              "krita_play_chunk",
                              KisAndroidFileProxy::getFileFromContentUri(file->absoluteFilePath()).toUtf8().data()));
#else
        new Mlt::Producer(*m_d->profile, "krita_play_chunk", file->absoluteFilePath().toUtf8().data()));
#endif
        if (producer->is_valid()) {
            m_d->canvasProducers[activeCanvas()] = producer;
        } else {
            // SANITY CHECK: Check that the MLT plugins and resources are where the program expects them to be.
            // HINT -- Check krita/main.cc's mlt environment variable setup for appimage.
            KIS_SAFE_ASSERT_RECOVER_NOOP(qEnvironmentVariableIsSet("MLT_REPOSITORY"));
            KIS_SAFE_ASSERT_RECOVER_NOOP(qEnvironmentVariableIsSet("MLT_PROFILES_PATH"));
            KIS_SAFE_ASSERT_RECOVER_NOOP(qEnvironmentVariableIsSet("MLT_PRESETS_PATH"));
            qDebug() << "Warning: Invalid MLT producer for file: " << ppVar(file->absoluteFilePath()) << " Falling back to audio-less playback.";
        }
    }
 
    KisImageAnimationInterface *animInterface = activeCanvas()->image()->animationInterface();
    QSharedPointer<Mlt::Producer> producer = m_d->canvasProducers[activeCanvas()];
    KIS_ASSERT(producer->is_valid());
    KIS_ASSERT(animInterface);
 
    producer->set("start_frame", animInterface->documentPlaybackRange().start());
    producer->set("end_frame", animInterface->documentPlaybackRange().end());
    producer->set("limit_enabled", false);
    producer->set("speed", m_d->playbackSpeed);
}

References KisPlaybackEngine::activeCanvas(), KisImage::animationInterface(), canvasDestroyed(), connect(), KisImageAnimationInterface::documentPlaybackRange(), KisTimeSpan::end(), KisAndroidFileProxy::getFileFromContentUri(), KisCanvas2::image(), KIS_ASSERT, KIS_SAFE_ASSERT_RECOVER_NOOP, m_d, ppVar, and KisTimeSpan::start().

sigChangeActiveCanvasFrame

void KisPlaybackEngineMLT::sigChangeActiveCanvasFrame ( int p_frame )

signal

supportsAudio

bool KisPlaybackEngineMLT::supportsAudio ( )

inlineoverrideslot

Definition at line 51 of file KisPlaybackEngineMLT.h.

{ return true; }

supportsVariablePlaybackSpeed

bool KisPlaybackEngineMLT::supportsVariablePlaybackSpeed ( )

inlineoverrideslot

Definition at line 52 of file KisPlaybackEngineMLT.h.

{ return true; }

throttledSetSpeed

void KisPlaybackEngineMLT::throttledSetSpeed ( const double speed )

protectedslot

throttledSetSpeed

Parameters

speed

Because MLT needs to be stopped and restarted to change playback speed we use this function to limit the frequency of speed change requests.

Definition at line 573 of file KisPlaybackEngineMLT.cpp.

{
    StopAndResume stopResume(m_d.data(), false);
    m_d->playbackSpeed = speed;
}

References m_d.

throttledShowFrame

void KisPlaybackEngineMLT::throttledShowFrame ( const int frame )

protectedslot

throttledShowFrame

Parameters

frame

In order to throttle calls from MLT to respect our playback mode, we need to redirect showFrame calls to this thread and enforce that we only allow MLT to show frames when we are in PULL mode.

Definition at line 549 of file KisPlaybackEngineMLT.cpp.

{
    if (activeCanvas() && activeCanvas()->animationState() &&
            m_d->activePlaybackMode() == PLAYBACK_PULL ) {
 
        if (m_d->frameStats.lastRenderedFrame < 0) {
            m_d->frameStats.timeSinceLastFrame.start();
        } else {
            const int droppedFrames = qMax(0, frame - m_d->frameStats.lastRenderedFrame - 1);
            m_d->frameStats.averageFrameDuration(m_d->frameStats.timeSinceLastFrame.restart());
            m_d->frameStats.droppedFramesCount(droppedFrames);
        }
        m_d->frameStats.lastRenderedFrame = frame;
 
        activeCanvas()->animationState()->showFrame(frame);
    }
 
    {
        QMutexLocker l(&m_d->frameWaitingInterface.renderingControlMutex);
        m_d->frameWaitingInterface.waitingForFrame = false;
        m_d->frameWaitingInterface.renderingWaitCondition.wakeAll();
    }
}

References KisPlaybackEngine::activeCanvas(), KisCanvas2::animationState(), m_d, PLAYBACK_PULL, and KisCanvasAnimationState::showFrame().

unsetCanvas

void KisPlaybackEngineMLT::unsetCanvas ( )

overrideprotectedslot

Definition at line 529 of file KisPlaybackEngineMLT.cpp.

                                       {
    setCanvas(nullptr);
}

References setCanvas().

Member Data Documentation

m_d

QScopedPointer<Private> KisPlaybackEngineMLT::m_d

private

Definition at line 104 of file KisPlaybackEngineMLT.h.

---

The documentation for this class was generated from the following files:

• libs/ui/KisPlaybackEngineMLT.h
• libs/ui/KisPlaybackEngineMLT.cpp